Oct 30 17:51:46.862734 ip-10-0-130-84 systemd[1]: Starting Kubernetes Kubelet... Oct 30 17:51:47.266342 ip-10-0-130-84 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:51:47.266342 ip-10-0-130-84 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 17:51:47.266342 ip-10-0-130-84 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:51:47.266342 ip-10-0-130-84 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:51:47.269742 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269652 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:51:47.269742 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269743 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:51:47.269823 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269756 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:51:47.269823 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269760 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:51:47.269823 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269764 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:51:47.269823 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269767 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:51:47.269823 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269770 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:51:47.269823 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269773 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:51:47.269823 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269775 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:51:47.269823 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269778 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:51:47.269823 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269781 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:51:47.269823 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269784 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:51:47.269823 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269787 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:51:47.269823 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269790 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:51:47.269823 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269793 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:51:47.269823 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269795 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:51:47.269823 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269798 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:51:47.269823 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269800 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:51:47.269823 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269803 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:51:47.269823 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269805 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:51:47.269823 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269808 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:51:47.269823 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269810 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:51:47.270290 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269813 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:51:47.270290 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269816 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:51:47.270290 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269818 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:51:47.270290 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269820 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:51:47.270290 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269823 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:51:47.270290 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269834 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:51:47.270290 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269837 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:51:47.270290 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269840 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:51:47.270290 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269842 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:51:47.270290 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269845 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:51:47.270290 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269847 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:51:47.270290 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269850 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:51:47.270290 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269852 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:51:47.270290 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269855 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:51:47.270290 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269857 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:51:47.270290 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269860 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:51:47.270290 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269863 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:51:47.270290 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269865 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:51:47.270290 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269868 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:51:47.270290 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269870 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:51:47.270769 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269873 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:51:47.270769 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269876 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:51:47.270769 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269878 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:51:47.270769 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269881 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:51:47.270769 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269884 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:51:47.270769 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269886 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:51:47.270769 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269889 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:51:47.270769 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269891 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:51:47.270769 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269894 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:51:47.270769 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269896 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:51:47.270769 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269899 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:51:47.270769 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269902 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:51:47.270769 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269905 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:51:47.270769 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269907 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:51:47.270769 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269910 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:51:47.270769 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269913 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:51:47.270769 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269915 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:51:47.270769 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269918 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:51:47.270769 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269920 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:51:47.271209 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269923 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:51:47.271209 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269926 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:51:47.271209 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269929 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:51:47.271209 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269931 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:51:47.271209 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269933 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:51:47.271209 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269936 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:51:47.271209 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269938 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:51:47.271209 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269941 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:51:47.271209 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269943 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:51:47.271209 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269947 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:51:47.271209 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269949 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:51:47.271209 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269952 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:51:47.271209 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269954 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:51:47.271209 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269957 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:51:47.271209 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269960 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:51:47.271209 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269962 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:51:47.271209 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269965 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:51:47.271209 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269970 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:51:47.271209 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269975 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:51:47.271705 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269979 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:51:47.271705 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269981 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:51:47.271705 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269984 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:51:47.271705 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269987 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:51:47.271705 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269990 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:51:47.271705 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269993 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:51:47.271705 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269995 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:51:47.271705 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.269998 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:51:47.271705 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270000 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:51:47.271705 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270003 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:51:47.271705 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270007 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:51:47.271705 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270011 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:51:47.271705 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270014 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:51:47.271705 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270017 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:51:47.271705 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270020 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:51:47.271705 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270023 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:51:47.271705 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270025 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:51:47.271705 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270028 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:51:47.271705 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270030 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:51:47.272152 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270033 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:51:47.272152 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270035 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:51:47.272152 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270038 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:51:47.272152 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270041 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:51:47.272152 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270043 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:51:47.272152 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270046 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:51:47.272152 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270048 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:51:47.272152 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270855 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:51:47.272152 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270862 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:51:47.272152 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270867 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:51:47.272152 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270871 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:51:47.272152 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270875 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:51:47.272152 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270878 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:51:47.272152 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270881 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:51:47.272152 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270884 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:51:47.272152 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270887 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:51:47.272152 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270890 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:51:47.272152 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270892 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:51:47.272152 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270895 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:51:47.272637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270898 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:51:47.272637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270901 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:51:47.272637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270903 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:51:47.272637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270906 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:51:47.272637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270908 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:51:47.272637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270911 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:51:47.272637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270913 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:51:47.272637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270917 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:51:47.272637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270920 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:51:47.272637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270922 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:51:47.272637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270925 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:51:47.272637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270927 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:51:47.272637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270930 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:51:47.272637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270932 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:51:47.272637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270935 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:51:47.272637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270937 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:51:47.272637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270940 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:51:47.272637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270942 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:51:47.272637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270945 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:51:47.273092 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270947 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:51:47.273092 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270950 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:51:47.273092 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270952 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:51:47.273092 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270955 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:51:47.273092 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270957 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:51:47.273092 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270960 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:51:47.273092 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270963 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:51:47.273092 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270966 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:51:47.273092 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270969 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:51:47.273092 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270972 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:51:47.273092 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270974 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:51:47.273092 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270977 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:51:47.273092 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270980 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:51:47.273092 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270982 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:51:47.273092 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270985 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:51:47.273092 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270988 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:51:47.273092 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270990 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:51:47.273092 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270993 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:51:47.273092 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270995 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:51:47.273092 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.270998 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:51:47.273693 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271000 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:51:47.273693 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271003 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:51:47.273693 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271006 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:51:47.273693 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271008 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:51:47.273693 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271011 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:51:47.273693 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271013 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:51:47.273693 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271015 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:51:47.273693 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271018 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:51:47.273693 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271020 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:51:47.273693 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271022 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:51:47.273693 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271025 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:51:47.273693 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271027 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:51:47.273693 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271029 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:51:47.273693 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271032 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:51:47.273693 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271036 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:51:47.273693 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271039 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:51:47.273693 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271043 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:51:47.273693 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271046 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:51:47.273693 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271049 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:51:47.274162 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271051 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:51:47.274162 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271054 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:51:47.274162 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271057 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:51:47.274162 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271059 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:51:47.274162 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271062 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:51:47.274162 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271065 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:51:47.274162 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271068 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:51:47.274162 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271070 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:51:47.274162 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271073 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:51:47.274162 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271075 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:51:47.274162 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271078 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:51:47.274162 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271080 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:51:47.274162 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271083 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:51:47.274162 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271085 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:51:47.274162 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271088 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:51:47.274162 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271090 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:51:47.274162 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271093 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:51:47.274162 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271095 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:51:47.274162 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271098 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:51:47.274637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271100 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:51:47.274637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271102 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:51:47.274637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271105 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:51:47.274637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271107 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:51:47.274637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271110 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:51:47.274637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271112 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:51:47.274637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271114 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:51:47.274637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271117 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:51:47.274637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271119 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:51:47.274637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271121 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:51:47.274637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271124 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:51:47.274637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271127 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:51:47.274637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271129 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:51:47.274637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271133 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:51:47.274637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271135 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:51:47.274637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271138 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:51:47.274637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271141 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:51:47.274637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271695 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:51:47.274637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271704 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:51:47.274637 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271709 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:51:47.275124 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271712 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:51:47.275124 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271714 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:51:47.275124 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271717 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:51:47.275124 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271720 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:51:47.275124 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271724 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:51:47.275124 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271726 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:51:47.275124 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271729 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:51:47.275124 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271731 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:51:47.275124 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271734 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:51:47.275124 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271736 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:51:47.275124 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271739 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:51:47.275124 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271741 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:51:47.275124 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271744 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:51:47.275124 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271746 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:51:47.275124 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271749 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:51:47.275124 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271751 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:51:47.275124 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271754 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:51:47.275124 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271757 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:51:47.275124 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271759 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:51:47.275587 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271762 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:51:47.275587 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271764 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:51:47.275587 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271767 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:51:47.275587 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271770 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:51:47.275587 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271773 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:51:47.275587 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271775 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:51:47.275587 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271778 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:51:47.275587 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271780 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:51:47.275587 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271784 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:51:47.275587 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271786 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:51:47.275587 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271789 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:51:47.275587 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271792 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:51:47.275587 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271795 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:51:47.275587 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271797 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:51:47.275587 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271800 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:51:47.275587 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271802 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:51:47.275587 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271804 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:51:47.275587 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271807 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:51:47.275587 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271809 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:51:47.275587 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271812 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:51:47.276059 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271815 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:51:47.276059 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271817 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:51:47.276059 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271819 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:51:47.276059 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271822 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:51:47.276059 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271824 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:51:47.276059 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271828 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:51:47.276059 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271830 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:51:47.276059 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271833 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:51:47.276059 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271836 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:51:47.276059 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271839 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:51:47.276059 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271841 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:51:47.276059 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271843 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:51:47.276059 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271846 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:51:47.276059 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271848 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:51:47.276059 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271851 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:51:47.276059 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271853 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:51:47.276059 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271856 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:51:47.276059 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271858 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:51:47.276059 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271861 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:51:47.276059 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271863 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:51:47.276578 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271866 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:51:47.276578 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271868 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:51:47.276578 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271871 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:51:47.276578 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271874 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:51:47.276578 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271877 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:51:47.276578 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271879 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:51:47.276578 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271882 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:51:47.276578 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271884 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:51:47.276578 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271887 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:51:47.276578 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271889 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:51:47.276578 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271892 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:51:47.276578 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271894 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:51:47.276578 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271898 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:51:47.276578 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271902 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:51:47.276578 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271904 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:51:47.276578 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271907 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:51:47.276578 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271909 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:51:47.276578 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271912 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:51:47.276578 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271914 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:51:47.277024 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271917 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:51:47.277024 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271922 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:51:47.277024 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271925 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:51:47.277024 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271928 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:51:47.277024 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271931 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:51:47.277024 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271933 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:51:47.277024 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271935 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:51:47.277024 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271938 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:51:47.277024 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271940 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:51:47.277024 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271943 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:51:47.277024 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271945 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:51:47.277024 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271947 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:51:47.277024 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271950 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:51:47.277024 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271952 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:51:47.277024 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271954 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:51:47.277024 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271957 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:51:47.277024 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271960 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:51:47.277024 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271963 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:51:47.277024 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271965 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:51:47.277024 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271968 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:51:47.277517 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271970 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:51:47.277517 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271972 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:51:47.277517 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271975 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:51:47.277517 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271977 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:51:47.277517 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.271979 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:51:47.277517 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272812 2579 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 17:51:47.277517 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272822 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 17:51:47.277517 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272828 2579 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 17:51:47.277517 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272833 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 17:51:47.277517 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272837 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 17:51:47.277517 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272840 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 17:51:47.277517 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272844 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 17:51:47.277517 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272849 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 17:51:47.277517 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272852 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 17:51:47.277517 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272855 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 17:51:47.277517 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272858 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 17:51:47.277517 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272861 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 17:51:47.277517 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272865 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 17:51:47.277517 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272867 2579 flags.go:64] FLAG: --cgroup-root="" Oct 30 17:51:47.277517 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272870 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 17:51:47.277517 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272873 2579 flags.go:64] FLAG: --client-ca-file="" Oct 30 17:51:47.277517 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272876 2579 flags.go:64] FLAG: --cloud-provider="external" Oct 30 17:51:47.277517 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272879 2579 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 17:51:47.277517 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272883 2579 flags.go:64] FLAG: --cluster-domain="" Oct 30 17:51:47.278111 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272886 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 17:51:47.278111 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272889 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 17:51:47.278111 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272892 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 17:51:47.278111 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272896 2579 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 17:51:47.278111 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272900 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 17:51:47.278111 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272904 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 17:51:47.278111 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272907 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 17:51:47.278111 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272910 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 17:51:47.278111 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272914 2579 flags.go:64] FLAG: --contention-profiling="false" Oct 30 17:51:47.278111 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272917 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 17:51:47.278111 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272920 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 17:51:47.278111 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272923 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 17:51:47.278111 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272926 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 17:51:47.278111 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272930 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 17:51:47.278111 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272933 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 17:51:47.278111 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272936 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 17:51:47.278111 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272939 2579 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 17:51:47.278111 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272942 2579 flags.go:64] FLAG: --enable-server="true" Oct 30 17:51:47.278111 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272944 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 17:51:47.278111 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272949 2579 flags.go:64] FLAG: --event-burst="100" Oct 30 17:51:47.278111 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272952 2579 flags.go:64] FLAG: --event-qps="50" Oct 30 17:51:47.278111 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272955 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 17:51:47.278111 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272959 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 17:51:47.278111 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272961 2579 flags.go:64] FLAG: --eviction-hard="" Oct 30 17:51:47.278712 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272966 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 17:51:47.278712 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272969 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 17:51:47.278712 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272972 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 17:51:47.278712 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272975 2579 flags.go:64] FLAG: --eviction-soft="" Oct 30 17:51:47.278712 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272978 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 17:51:47.278712 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272981 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 17:51:47.278712 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272983 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 17:51:47.278712 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272986 2579 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 17:51:47.278712 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272989 2579 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 17:51:47.278712 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272992 2579 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 17:51:47.278712 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.272997 2579 flags.go:64] FLAG: --feature-gates="" Oct 30 17:51:47.278712 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273001 2579 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 17:51:47.278712 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273004 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 17:51:47.278712 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273008 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 17:51:47.278712 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273011 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 17:51:47.278712 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273017 2579 flags.go:64] FLAG: --healthz-port="10248" Oct 30 17:51:47.278712 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273020 2579 flags.go:64] FLAG: --help="false" Oct 30 17:51:47.278712 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273023 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-130-84.ec2.internal" Oct 30 17:51:47.278712 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273026 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 17:51:47.278712 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273029 2579 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 17:51:47.278712 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273032 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 17:51:47.278712 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273035 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 17:51:47.278712 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273039 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 17:51:47.279264 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273042 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 17:51:47.279264 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273045 2579 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 17:51:47.279264 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273048 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 17:51:47.279264 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273051 2579 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 17:51:47.279264 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273054 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 17:51:47.279264 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273057 2579 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 17:51:47.279264 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273060 2579 flags.go:64] FLAG: --kube-reserved="" Oct 30 17:51:47.279264 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273063 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 17:51:47.279264 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273066 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 17:51:47.279264 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273069 2579 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 17:51:47.279264 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273072 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 17:51:47.279264 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273075 2579 flags.go:64] FLAG: --lock-file="" Oct 30 17:51:47.279264 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273078 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 17:51:47.279264 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273081 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 17:51:47.279264 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273084 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 17:51:47.279264 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273089 2579 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 17:51:47.279264 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273092 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 17:51:47.279264 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273095 2579 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 17:51:47.279264 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273098 2579 flags.go:64] FLAG: --logging-format="text" Oct 30 17:51:47.279264 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273103 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 17:51:47.279264 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273106 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 17:51:47.279264 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273110 2579 flags.go:64] FLAG: --manifest-url="" Oct 30 17:51:47.279264 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273113 2579 flags.go:64] FLAG: --manifest-url-header="" Oct 30 17:51:47.279264 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273121 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 17:51:47.279264 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273126 2579 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 17:51:47.279884 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273130 2579 flags.go:64] FLAG: --max-pods="110" Oct 30 17:51:47.279884 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273133 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 17:51:47.279884 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273136 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 17:51:47.279884 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273139 2579 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 17:51:47.279884 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273142 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 17:51:47.279884 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273145 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 17:51:47.279884 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273148 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 17:51:47.279884 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273151 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 17:51:47.279884 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273158 2579 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 17:51:47.279884 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273161 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 17:51:47.279884 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273164 2579 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 17:51:47.279884 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273167 2579 flags.go:64] FLAG: --pod-cidr="" Oct 30 17:51:47.279884 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273169 2579 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 17:51:47.279884 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273172 2579 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 17:51:47.279884 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273175 2579 flags.go:64] FLAG: --pods-per-core="0" Oct 30 17:51:47.279884 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273178 2579 flags.go:64] FLAG: --port="10250" Oct 30 17:51:47.279884 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273181 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 17:51:47.279884 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273184 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c358448deb9909bd" Oct 30 17:51:47.279884 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273187 2579 flags.go:64] FLAG: --qos-reserved="" Oct 30 17:51:47.279884 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273190 2579 flags.go:64] FLAG: --read-only-port="10255" Oct 30 17:51:47.279884 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273193 2579 flags.go:64] FLAG: --register-node="true" Oct 30 17:51:47.279884 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273196 2579 flags.go:64] FLAG: --register-with-taints="" Oct 30 17:51:47.279884 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273200 2579 flags.go:64] FLAG: --registry-burst="10" Oct 30 17:51:47.279884 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273203 2579 flags.go:64] FLAG: --registry-qps="5" Oct 30 17:51:47.279884 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273205 2579 flags.go:64] FLAG: --reserved-cpus="" Oct 30 17:51:47.279884 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273208 2579 flags.go:64] FLAG: --reserved-memory="" Oct 30 17:51:47.280508 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273214 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 17:51:47.280508 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273217 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 17:51:47.280508 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273220 2579 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 17:51:47.280508 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273223 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 17:51:47.280508 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273226 2579 flags.go:64] FLAG: --runonce="false" Oct 30 17:51:47.280508 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273231 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 17:51:47.280508 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273234 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 17:51:47.280508 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273237 2579 flags.go:64] FLAG: --seccomp-default="false" Oct 30 17:51:47.280508 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273240 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 17:51:47.280508 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273242 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 17:51:47.280508 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273246 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 17:51:47.280508 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273249 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 17:51:47.280508 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273252 2579 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 17:51:47.280508 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273255 2579 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 17:51:47.280508 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273258 2579 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 17:51:47.280508 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273261 2579 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 17:51:47.280508 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273264 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 17:51:47.280508 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273267 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 17:51:47.280508 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273269 2579 flags.go:64] FLAG: --system-cgroups="" Oct 30 17:51:47.280508 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273272 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 17:51:47.280508 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273277 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 17:51:47.280508 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273280 2579 flags.go:64] FLAG: --tls-cert-file="" Oct 30 17:51:47.280508 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273283 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 17:51:47.280508 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273287 2579 flags.go:64] FLAG: --tls-min-version="" Oct 30 17:51:47.280508 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273290 2579 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 17:51:47.281125 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273293 2579 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 17:51:47.281125 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273296 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 17:51:47.281125 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273299 2579 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 17:51:47.281125 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273302 2579 flags.go:64] FLAG: --v="2" Oct 30 17:51:47.281125 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273306 2579 flags.go:64] FLAG: --version="false" Oct 30 17:51:47.281125 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273310 2579 flags.go:64] FLAG: --vmodule="" Oct 30 17:51:47.281125 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273315 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 17:51:47.281125 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273331 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 17:51:47.281125 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273472 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:51:47.281125 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273475 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:51:47.281125 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273478 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:51:47.281125 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273482 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:51:47.281125 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273486 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:51:47.281125 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273489 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:51:47.281125 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273491 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:51:47.281125 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273496 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:51:47.281125 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273499 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:51:47.281125 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273502 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:51:47.281125 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273504 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:51:47.281125 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273507 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:51:47.281125 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273510 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:51:47.281125 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273512 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:51:47.281675 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273515 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:51:47.281675 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273517 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:51:47.281675 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273520 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:51:47.281675 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273523 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:51:47.281675 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273526 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:51:47.281675 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273528 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:51:47.281675 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273531 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:51:47.281675 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273533 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:51:47.281675 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273536 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:51:47.281675 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273538 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:51:47.281675 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273541 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:51:47.281675 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273544 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:51:47.281675 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273546 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:51:47.281675 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273548 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:51:47.281675 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273551 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:51:47.281675 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273553 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:51:47.281675 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273556 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:51:47.281675 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273560 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:51:47.281675 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273562 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:51:47.282139 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273565 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:51:47.282139 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273568 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:51:47.282139 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273570 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:51:47.282139 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273574 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:51:47.282139 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273576 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:51:47.282139 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273581 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:51:47.282139 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273585 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:51:47.282139 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273588 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:51:47.282139 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273590 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:51:47.282139 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273593 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:51:47.282139 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273595 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:51:47.282139 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273598 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:51:47.282139 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273601 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:51:47.282139 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273603 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:51:47.282139 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273606 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:51:47.282139 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273608 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:51:47.282139 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273611 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:51:47.282139 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273613 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:51:47.282638 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273616 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:51:47.282638 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273618 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:51:47.282638 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273621 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:51:47.282638 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273623 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:51:47.282638 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273626 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:51:47.282638 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273628 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:51:47.282638 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273631 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:51:47.282638 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273633 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:51:47.282638 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273635 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:51:47.282638 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273638 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:51:47.282638 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273640 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:51:47.282638 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273643 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:51:47.282638 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273647 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:51:47.282638 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273649 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:51:47.282638 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273652 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:51:47.282638 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273654 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:51:47.282638 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273657 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:51:47.282638 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273660 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:51:47.282638 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273663 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:51:47.282638 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273665 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:51:47.283113 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273672 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:51:47.283113 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273675 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:51:47.283113 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273678 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:51:47.283113 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273680 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:51:47.283113 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273683 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:51:47.283113 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273685 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:51:47.283113 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273688 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:51:47.283113 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273690 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:51:47.283113 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273694 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:51:47.283113 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273696 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:51:47.283113 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273699 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:51:47.283113 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273702 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:51:47.283113 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273704 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:51:47.283113 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273707 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:51:47.283113 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273710 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:51:47.283113 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273713 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:51:47.283113 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273716 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:51:47.283113 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273718 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:51:47.283113 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273721 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:51:47.283113 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273723 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:51:47.284078 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273726 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:51:47.284078 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273728 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:51:47.284078 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273731 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:51:47.284078 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273733 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:51:47.284078 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273737 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:51:47.284078 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273739 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:51:47.284078 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273743 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:51:47.284078 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273753 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:51:47.284078 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273756 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:51:47.284078 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273761 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:51:47.284078 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273763 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:51:47.284078 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273766 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:51:47.284078 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273768 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:51:47.284078 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273773 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:51:47.284078 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273776 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:51:47.284078 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.273803 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:51:47.284482 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273841 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:51:47.284482 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273845 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:51:47.284482 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273848 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:51:47.284482 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273852 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:51:47.284482 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273854 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:51:47.284482 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273857 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:51:47.284482 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273860 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:51:47.284482 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273862 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:51:47.284482 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273865 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:51:47.284482 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273868 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:51:47.284482 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273871 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:51:47.284482 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273874 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:51:47.284482 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273876 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:51:47.284482 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273879 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:51:47.284482 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273883 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:51:47.284482 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273886 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:51:47.284482 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273889 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:51:47.284482 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273892 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:51:47.284482 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273894 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:51:47.284482 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273897 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:51:47.284942 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273899 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:51:47.284942 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273903 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:51:47.284942 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273906 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:51:47.284942 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273908 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:51:47.284942 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273911 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:51:47.284942 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273915 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:51:47.284942 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273917 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:51:47.284942 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273920 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:51:47.284942 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273922 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:51:47.284942 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273925 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:51:47.284942 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273928 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:51:47.284942 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273930 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:51:47.284942 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273933 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:51:47.284942 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273935 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:51:47.284942 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273938 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:51:47.284942 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273940 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:51:47.284942 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273943 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:51:47.284942 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273945 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:51:47.284942 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273948 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:51:47.284942 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273950 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:51:47.285430 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273953 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:51:47.285430 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273955 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:51:47.285430 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273958 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:51:47.285430 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273960 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:51:47.285430 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273963 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:51:47.285430 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273967 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:51:47.285430 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273970 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:51:47.285430 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273973 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:51:47.285430 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273976 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:51:47.285430 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273978 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:51:47.285430 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273981 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:51:47.285430 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273983 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:51:47.285430 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273986 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:51:47.285430 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273989 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:51:47.285430 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273993 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:51:47.285430 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273995 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:51:47.285430 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.273998 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:51:47.285430 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274001 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:51:47.285866 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274004 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:51:47.285866 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274006 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:51:47.285866 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274009 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:51:47.285866 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274012 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:51:47.285866 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274014 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:51:47.285866 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274017 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:51:47.285866 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274020 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:51:47.285866 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274022 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:51:47.285866 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274025 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:51:47.285866 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274028 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:51:47.285866 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274030 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:51:47.285866 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274033 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:51:47.285866 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274035 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:51:47.285866 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274037 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:51:47.285866 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274040 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:51:47.285866 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274042 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:51:47.285866 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274045 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:51:47.285866 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274047 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:51:47.285866 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274050 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:51:47.285866 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274052 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:51:47.286413 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274054 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:51:47.286413 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274057 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:51:47.286413 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274059 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:51:47.286413 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274062 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:51:47.286413 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274064 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:51:47.286413 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274067 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:51:47.286413 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274069 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:51:47.286413 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274073 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:51:47.286413 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274075 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:51:47.286413 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274078 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:51:47.286413 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274080 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:51:47.286413 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274084 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:51:47.286413 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274087 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:51:47.286413 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274090 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:51:47.286413 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274092 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:51:47.286413 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274095 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:51:47.286413 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274097 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:51:47.286413 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274100 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:51:47.286413 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274103 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:51:47.286413 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274105 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:51:47.286885 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274108 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:51:47.286885 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274110 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:51:47.286885 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274113 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:51:47.286885 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274115 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:51:47.286885 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274119 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:51:47.286885 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274121 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:51:47.286885 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274124 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:51:47.286885 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.274126 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:51:47.286885 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.281474 2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 17:51:47.286885 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.281491 2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 17:51:47.286885 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.281503 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 17:51:47.286885 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.281508 2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 17:51:47.286885 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281567 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:51:47.286885 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281572 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:51:47.286885 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281575 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:51:47.286885 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281578 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:51:47.286885 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281581 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:51:47.286885 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281583 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:51:47.286885 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281586 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:51:47.286885 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281588 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:51:47.287399 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281591 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:51:47.287399 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281593 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:51:47.287399 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281596 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:51:47.287399 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281598 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:51:47.287399 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281601 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:51:47.287399 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281603 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:51:47.287399 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281606 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:51:47.287399 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281608 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:51:47.287399 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281611 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:51:47.287399 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281613 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:51:47.287399 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281616 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:51:47.287399 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281620 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:51:47.287399 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281624 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:51:47.287399 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281628 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:51:47.287399 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281631 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:51:47.287399 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281633 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:51:47.287399 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281636 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:51:47.287399 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281639 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:51:47.287826 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281641 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:51:47.287826 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281644 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:51:47.287826 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281646 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:51:47.287826 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281650 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:51:47.287826 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281653 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:51:47.287826 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281655 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:51:47.287826 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281658 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:51:47.287826 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281661 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:51:47.287826 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281663 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:51:47.287826 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281667 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:51:47.287826 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281671 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:51:47.287826 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281674 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:51:47.287826 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281677 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:51:47.287826 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281680 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:51:47.287826 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281683 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:51:47.287826 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281686 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:51:47.287826 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281689 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:51:47.287826 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281692 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:51:47.287826 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281694 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:51:47.288295 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281697 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:51:47.288295 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281699 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:51:47.288295 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281702 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:51:47.288295 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281704 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:51:47.288295 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281707 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:51:47.288295 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281709 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:51:47.288295 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281713 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:51:47.288295 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281716 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:51:47.288295 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281718 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:51:47.288295 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281721 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:51:47.288295 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281723 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:51:47.288295 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281725 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:51:47.288295 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281728 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:51:47.288295 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281730 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:51:47.288295 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281733 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:51:47.288295 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281735 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:51:47.288295 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281739 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:51:47.288295 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281741 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:51:47.288295 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281744 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:51:47.288832 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281760 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:51:47.288832 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281763 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:51:47.288832 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281765 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:51:47.288832 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281768 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:51:47.288832 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281771 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:51:47.288832 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281773 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:51:47.288832 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281776 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:51:47.288832 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281778 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:51:47.288832 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281781 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:51:47.288832 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281783 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:51:47.288832 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281786 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:51:47.288832 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281788 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:51:47.288832 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281791 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:51:47.288832 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281793 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:51:47.288832 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281795 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:51:47.288832 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281798 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:51:47.288832 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281800 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:51:47.288832 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281803 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:51:47.288832 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281805 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:51:47.288832 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281807 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:51:47.289299 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281815 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:51:47.289299 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281818 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:51:47.289299 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281820 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:51:47.289299 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281823 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:51:47.289299 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281826 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:51:47.289299 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281828 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:51:47.289299 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281831 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:51:47.289299 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281833 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:51:47.289299 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281835 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:51:47.289299 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281839 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:51:47.289299 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281842 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:51:47.289299 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281844 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:51:47.289299 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281847 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:51:47.289299 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281850 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:51:47.289299 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281852 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:51:47.289299 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281854 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:51:47.289299 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281857 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:51:47.289299 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281860 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:51:47.289299 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281862 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:51:47.289299 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281865 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:51:47.289795 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281867 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:51:47.289795 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.281870 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:51:47.289795 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.281898 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:51:47.289795 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282039 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:51:47.289795 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282044 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:51:47.289795 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282048 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:51:47.289795 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282051 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:51:47.289795 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282054 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:51:47.289795 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282056 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:51:47.289795 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282059 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:51:47.289795 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282062 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:51:47.289795 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282064 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:51:47.289795 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282067 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:51:47.289795 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282070 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:51:47.289795 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282072 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:51:47.289795 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282075 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:51:47.290169 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282077 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:51:47.290169 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282080 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:51:47.290169 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282127 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:51:47.290169 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282131 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:51:47.290169 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282133 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:51:47.290169 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282137 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:51:47.290169 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282140 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:51:47.290169 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282143 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:51:47.290169 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282145 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:51:47.290169 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282148 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:51:47.290169 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282151 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:51:47.290169 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282153 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:51:47.290169 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282156 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:51:47.290169 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282158 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:51:47.290169 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282161 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:51:47.290169 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282164 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:51:47.290169 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282166 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:51:47.290169 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282168 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:51:47.290169 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282171 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:51:47.290671 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282173 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:51:47.290671 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282175 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:51:47.290671 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282178 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:51:47.290671 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282181 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:51:47.290671 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282184 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:51:47.290671 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282188 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:51:47.290671 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282191 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:51:47.290671 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282193 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:51:47.290671 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282215 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:51:47.290671 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282218 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:51:47.290671 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282222 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:51:47.290671 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282225 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:51:47.290671 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282228 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:51:47.290671 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282231 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:51:47.290671 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282233 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:51:47.290671 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282237 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:51:47.290671 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282241 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:51:47.290671 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282244 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:51:47.290671 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282246 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:51:47.291104 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282249 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:51:47.291104 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282252 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:51:47.291104 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282254 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:51:47.291104 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282257 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:51:47.291104 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282259 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:51:47.291104 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282262 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:51:47.291104 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282265 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:51:47.291104 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282268 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:51:47.291104 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282270 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:51:47.291104 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282273 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:51:47.291104 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282275 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:51:47.291104 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282277 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:51:47.291104 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282280 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:51:47.291104 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282283 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:51:47.291104 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282285 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:51:47.291104 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282288 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:51:47.291104 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282290 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:51:47.291104 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282293 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:51:47.291104 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282295 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:51:47.291104 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282298 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:51:47.291662 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282300 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:51:47.291662 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282302 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:51:47.291662 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282305 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:51:47.291662 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282307 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:51:47.291662 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282309 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:51:47.291662 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282313 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:51:47.291662 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282315 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:51:47.291662 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282336 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:51:47.291662 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282339 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:51:47.291662 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282342 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:51:47.291662 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282345 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:51:47.291662 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282348 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:51:47.291662 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282350 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:51:47.291662 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282353 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:51:47.291662 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282355 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:51:47.291662 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282358 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:51:47.291662 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282360 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:51:47.291662 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282363 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:51:47.291662 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282365 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:51:47.291662 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282368 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:51:47.292141 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282370 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:51:47.292141 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282373 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:51:47.292141 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282375 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:51:47.292141 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282378 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:51:47.292141 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282380 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:51:47.292141 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282383 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:51:47.292141 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282385 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:51:47.292141 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282388 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:51:47.292141 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282390 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:51:47.292141 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282393 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:51:47.292141 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282395 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:51:47.292141 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282398 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:51:47.292141 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282400 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:51:47.292141 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282403 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:51:47.292141 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282405 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:51:47.292141 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.282430 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:51:47.292529 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282466 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:51:47.292529 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282470 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:51:47.292529 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282476 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:51:47.292529 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282479 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:51:47.292529 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282481 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:51:47.292529 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282484 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:51:47.292529 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282486 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:51:47.292529 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282489 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:51:47.292529 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282492 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:51:47.292529 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282494 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:51:47.292529 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282497 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:51:47.292529 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282500 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:51:47.292529 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282502 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:51:47.292529 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282505 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:51:47.292529 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282507 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:51:47.292529 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282510 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:51:47.292529 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282512 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:51:47.292529 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282514 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:51:47.292529 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282517 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:51:47.293260 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282519 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:51:47.293260 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282522 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:51:47.293260 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282524 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:51:47.293260 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282526 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:51:47.293260 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282529 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:51:47.293260 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282531 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:51:47.293260 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282534 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:51:47.293260 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282537 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:51:47.293260 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282540 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:51:47.293260 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282544 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:51:47.293260 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282547 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:51:47.293260 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282549 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:51:47.293260 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282552 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:51:47.293260 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282555 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:51:47.293260 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282557 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:51:47.293260 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282560 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:51:47.293260 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282563 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:51:47.293260 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282565 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:51:47.293260 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282568 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:51:47.294008 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282571 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:51:47.294008 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282576 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:51:47.294008 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282579 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:51:47.294008 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282582 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:51:47.294008 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282585 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:51:47.294008 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282587 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:51:47.294008 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282590 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:51:47.294008 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282592 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:51:47.294008 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282595 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:51:47.294008 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282597 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:51:47.294008 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282600 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:51:47.294008 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282602 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:51:47.294008 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282605 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:51:47.294008 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282607 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:51:47.294008 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282610 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:51:47.294008 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282612 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:51:47.294008 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282614 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:51:47.294008 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282617 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:51:47.294008 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282619 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:51:47.294634 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282622 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:51:47.294634 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282624 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:51:47.294634 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282627 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:51:47.294634 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282629 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:51:47.294634 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282633 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:51:47.294634 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282635 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:51:47.294634 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282637 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:51:47.294634 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282640 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:51:47.294634 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282642 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:51:47.294634 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282645 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:51:47.294634 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282647 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:51:47.294634 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282650 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:51:47.294634 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282653 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:51:47.294634 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282656 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:51:47.294634 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282658 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:51:47.294634 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282661 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:51:47.294634 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282664 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:51:47.294634 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282667 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:51:47.294634 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282670 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:51:47.294634 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282672 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:51:47.295121 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282675 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:51:47.295121 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282677 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:51:47.295121 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282679 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:51:47.295121 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282682 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:51:47.295121 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282684 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:51:47.295121 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282687 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:51:47.295121 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282689 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:51:47.295121 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282691 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:51:47.295121 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282693 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:51:47.295121 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282696 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:51:47.295121 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282698 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:51:47.295121 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282701 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:51:47.295121 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282703 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:51:47.295121 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282706 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:51:47.295121 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282708 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:51:47.295121 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282712 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:51:47.295121 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282714 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:51:47.295121 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282716 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:51:47.295121 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282719 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:51:47.295121 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282721 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:51:47.295688 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282724 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:51:47.295688 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282726 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:51:47.295688 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282728 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:51:47.295688 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282731 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:51:47.295688 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282733 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:51:47.295688 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282736 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:51:47.295688 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282741 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:51:47.295688 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282743 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:51:47.295688 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:47.282746 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:51:47.295688 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.282799 2579 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 17:51:47.295688 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.285019 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 17:51:47.295688 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.285992 2579 server.go:1014] "Starting client certificate rotation" Oct 30 17:51:47.295688 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.286096 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 17:51:47.295688 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.286870 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 17:51:47.295688 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.287766 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:51:47.296493 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.296475 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:51:47.307964 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.307947 2579 log.go:25] "Validated CRI v1 runtime API" Oct 30 17:51:47.311225 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.311207 2579 log.go:25] "Validated CRI v1 image API" Oct 30 17:51:47.312145 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.312106 2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 17:51:47.316620 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.316587 2579 fs.go:135] Filesystem UUIDs: map[6058e3e4-674f-4c8d-a77f-eb4f26f262d9:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ed7cf5a5-e0a5-4f68-9534-cfa0f67ed3bb:/dev/nvme0n1p4] Oct 30 17:51:47.316757 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.316616 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 17:51:47.322245 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.322117 2579 manager.go:217] Machine: {Timestamp:2026-10-30 17:51:47.320855013 +0000 UTC m=+0.339812181 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:2977970 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2869ed0b07fc42e7e792d2d96458d6 SystemUUID:ec2869ed-0b07-fc42-e7e7-92d2d96458d6 BootID:9f1dd574-157a-4a3c-be46-bc2b290da3f3 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d6:b9:1c:57 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d6:b9:1c:57 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:06:bb:4b:53:1b:e7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 17:51:47.322699 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.322689 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 17:51:47.322813 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.322795 2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 17:51:47.323993 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.323966 2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 17:51:47.324135 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.323994 2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-84.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 17:51:47.324661 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.324650 2579 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 17:51:47.324693 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.324663 2579 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 17:51:47.324693 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.324690 2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:51:47.324741 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.324704 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:51:47.325572 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.325562 2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 17:51:47.328412 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.328399 2579 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 17:51:47.328546 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.328537 2579 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 17:51:47.329298 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.329287 2579 kubelet.go:493] "Attempting to sync node with API server" Oct 30 17:51:47.329368 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.329306 2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 17:51:47.329368 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.329338 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 17:51:47.329368 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.329351 2579 kubelet.go:405] "Adding apiserver pod source" Oct 30 17:51:47.329368 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.329364 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 17:51:47.329792 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.329769 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:51:47.332732 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.332705 2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 17:51:47.337985 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.337968 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:51:47.338890 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.338874 2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 17:51:47.338972 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.338916 2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 17:51:47.339122 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.339108 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 17:51:47.339188 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.339129 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 17:51:47.339188 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.339139 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 17:51:47.339188 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.339148 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 17:51:47.339188 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.339161 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 17:51:47.339188 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.339170 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 17:51:47.339698 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.339688 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 17:51:47.339759 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.339702 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 17:51:47.339759 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.339713 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 17:51:47.339759 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.339723 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 17:51:47.339759 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.339747 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 17:51:47.339932 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.339760 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 17:51:47.340515 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.340503 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 17:51:47.340576 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.340520 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 17:51:47.344673 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.344657 2579 server.go:1263] "Started kubelet" Oct 30 17:51:47.344753 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.344706 2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 17:51:47.344840 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.344787 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 17:51:47.344907 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.344860 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 17:51:47.345501 ip-10-0-130-84 systemd[1]: Started Kubernetes Kubelet. Oct 30 17:51:47.348188 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.348132 2579 server.go:317] "Adding debug handlers to kubelet server" Oct 30 17:51:47.351100 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.351020 2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 17:51:47.353110 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:47.353089 2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 17:51:47.355229 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.355210 2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 17:51:47.355315 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.355229 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 17:51:47.355315 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:47.355282 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-84.ec2.internal\" not found" Oct 30 17:51:47.355971 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.355955 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 17:51:47.356054 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.355956 2579 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 17:51:47.356054 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.355987 2579 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 17:51:47.356054 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.356025 2579 factory.go:55] Registering systemd factory Oct 30 17:51:47.356054 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.356045 2579 factory.go:223] Registration of the systemd container factory successfully Oct 30 17:51:47.356203 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.356072 2579 reconstruct.go:97] "Volume reconstruction finished" Oct 30 17:51:47.356203 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.356081 2579 reconciler.go:29] "Reconciler: start to sync state" Oct 30 17:51:47.356532 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.356497 2579 factory.go:153] Registering CRI-O factory Oct 30 17:51:47.356532 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.356515 2579 factory.go:223] Registration of the crio container factory successfully Oct 30 17:51:47.356808 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.356568 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 17:51:47.356808 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.356606 2579 factory.go:103] Registering Raw factory Oct 30 17:51:47.356808 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.356639 2579 manager.go:1199] Started watching for new ooms in manager Oct 30 17:51:47.357868 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.357848 2579 manager.go:319] Starting recovery of all containers Oct 30 17:51:47.368157 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.368143 2579 manager.go:324] Recovery completed Oct 30 17:51:47.372260 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.372245 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:51:47.373840 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.373816 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-84.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:51:47.373919 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.373852 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-84.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:51:47.373919 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.373866 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-84.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:51:47.374413 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.374401 2579 cpu_manager.go:226] "Starting" policy="none" Oct 30 17:51:47.374472 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.374414 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:51:47.374472 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.374414 2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 17:51:47.374472 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.374450 2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 17:51:47.375068 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.375053 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-84.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:51:47.375122 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.375085 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-84.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:51:47.375122 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.375100 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-84.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:51:47.375874 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.375861 2579 policy_none.go:50] "Start" Oct 30 17:51:47.375938 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.375887 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 17:51:47.375938 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.375901 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 17:51:47.377173 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.377161 2579 policy_none.go:44] "Start" Oct 30 17:51:47.381776 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.381742 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-84.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 17:51:47.381776 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:47.381756 2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-84.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 17:51:47.382689 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:47.381734 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-84.ec2.internal.18e360121198101a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-84.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-84.ec2.internal,},FirstTimestamp:2026-10-30 17:51:47.344445466 +0000 UTC m=+0.363402632,LastTimestamp:2026-10-30 17:51:47.344445466 +0000 UTC m=+0.363402632,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-84.ec2.internal,}" Oct 30 17:51:47.402523 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.402504 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 17:51:47.402620 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.402602 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xpjjw" Oct 30 17:51:47.406505 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:47.406262 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-84.ec2.internal.18e360121358867a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-84.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-84.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-84.ec2.internal,},FirstTimestamp:2026-10-30 17:51:47.373835898 +0000 UTC m=+0.392793063,LastTimestamp:2026-10-30 17:51:47.373835898 +0000 UTC m=+0.392793063,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-84.ec2.internal,}" Oct 30 17:51:47.407003 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:47.406980 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 17:51:47.407003 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:47.406981 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-84.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 17:51:47.407102 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:47.406982 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 17:51:47.412235 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.412217 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xpjjw" Oct 30 17:51:47.415913 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.415885 2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 17:51:47.418247 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.418235 2579 manager.go:341] "Starting Device Plugin manager" Oct 30 17:51:47.418302 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:47.418265 2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 17:51:47.418302 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.418274 2579 server.go:87] "Starting device plugin registration server" Oct 30 17:51:47.418543 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.418526 2579 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 17:51:47.418631 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.418542 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 17:51:47.418705 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.418692 2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 17:51:47.418768 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.418758 2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 17:51:47.418768 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.418764 2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 17:51:47.419417 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:47.419140 2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 17:51:47.419417 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:47.419196 2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-84.ec2.internal\" not found" Oct 30 17:51:47.426441 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:47.426375 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-84.ec2.internal.18e360121358dfbc default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-84.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-84.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-84.ec2.internal,},FirstTimestamp:2026-10-30 17:51:47.373858748 +0000 UTC m=+0.392815918,LastTimestamp:2026-10-30 17:51:47.373858748 +0000 UTC m=+0.392815918,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-84.ec2.internal,}" Oct 30 17:51:47.496669 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.496633 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 17:51:47.497911 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.497896 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 17:51:47.498002 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.497919 2579 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 17:51:47.498002 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.497943 2579 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 17:51:47.498002 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:47.497981 2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 17:51:47.513959 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.513887 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:51:47.518940 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.518924 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:51:47.519867 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.519851 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-84.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:51:47.519959 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.519878 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-84.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:51:47.519959 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.519889 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-84.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:51:47.519959 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.519918 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-84.ec2.internal" Oct 30 17:51:47.541501 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.541481 2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-84.ec2.internal" Oct 30 17:51:47.541590 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:47.541505 2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-84.ec2.internal\": node \"ip-10-0-130-84.ec2.internal\" not found" Oct 30 17:51:47.588857 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:47.588832 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-84.ec2.internal\" not found" Oct 30 17:51:47.598742 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.598721 2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-84.ec2.internal"] Oct 30 17:51:47.598793 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.598781 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:51:47.599681 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.599668 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-84.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:51:47.599749 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.599695 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-84.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:51:47.599749 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.599704 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-84.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:51:47.601089 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.601077 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:51:47.601228 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.601212 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal" Oct 30 17:51:47.601274 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.601245 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:51:47.601811 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.601794 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-84.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:51:47.601811 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.601808 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-84.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:51:47.601907 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.601825 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-84.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:51:47.601907 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.601825 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-84.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:51:47.601907 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.601836 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-84.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:51:47.601907 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.601843 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-84.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:51:47.603011 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.602996 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-84.ec2.internal" Oct 30 17:51:47.603061 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.603023 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:51:47.603743 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.603730 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-84.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:51:47.603819 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.603755 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-84.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:51:47.603819 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.603767 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-84.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:51:47.625906 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:47.625886 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-84.ec2.internal\" not found" node="ip-10-0-130-84.ec2.internal" Oct 30 17:51:47.628389 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:47.628372 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-84.ec2.internal\" not found" node="ip-10-0-130-84.ec2.internal" Oct 30 17:51:47.657050 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.657014 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5ed893d4ed91b2f423e686d98f2bd57e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal\" (UID: \"5ed893d4ed91b2f423e686d98f2bd57e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal" Oct 30 17:51:47.657050 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.657057 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ed893d4ed91b2f423e686d98f2bd57e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal\" (UID: \"5ed893d4ed91b2f423e686d98f2bd57e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal" Oct 30 17:51:47.657205 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.657073 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/da45dfd37aee5c345465266021945ad3-config\") pod \"kube-apiserver-proxy-ip-10-0-130-84.ec2.internal\" (UID: \"da45dfd37aee5c345465266021945ad3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-84.ec2.internal" Oct 30 17:51:47.689412 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:47.689390 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-84.ec2.internal\" not found" Oct 30 17:51:47.757703 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.757674 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5ed893d4ed91b2f423e686d98f2bd57e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal\" (UID: \"5ed893d4ed91b2f423e686d98f2bd57e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal" Oct 30 17:51:47.757795 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.757707 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ed893d4ed91b2f423e686d98f2bd57e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal\" (UID: \"5ed893d4ed91b2f423e686d98f2bd57e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal" Oct 30 17:51:47.757795 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.757724 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/da45dfd37aee5c345465266021945ad3-config\") pod \"kube-apiserver-proxy-ip-10-0-130-84.ec2.internal\" (UID: \"da45dfd37aee5c345465266021945ad3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-84.ec2.internal" Oct 30 17:51:47.757795 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.757788 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/da45dfd37aee5c345465266021945ad3-config\") pod \"kube-apiserver-proxy-ip-10-0-130-84.ec2.internal\" (UID: \"da45dfd37aee5c345465266021945ad3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-84.ec2.internal" Oct 30 17:51:47.757910 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.757792 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ed893d4ed91b2f423e686d98f2bd57e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal\" (UID: \"5ed893d4ed91b2f423e686d98f2bd57e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal" Oct 30 17:51:47.757910 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.757812 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5ed893d4ed91b2f423e686d98f2bd57e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal\" (UID: \"5ed893d4ed91b2f423e686d98f2bd57e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal" Oct 30 17:51:47.790249 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:47.790191 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-84.ec2.internal\" not found" Oct 30 17:51:47.891312 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:47.891275 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-84.ec2.internal\" not found" Oct 30 17:51:47.929737 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.929690 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal" Oct 30 17:51:47.931835 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:47.931819 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-84.ec2.internal" Oct 30 17:51:47.991759 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:47.991733 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-84.ec2.internal\" not found" Oct 30 17:51:48.092377 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:48.092268 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-84.ec2.internal\" not found" Oct 30 17:51:48.192783 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:48.192746 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-84.ec2.internal\" not found" Oct 30 17:51:48.286358 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:48.286302 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 17:51:48.286831 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:48.286474 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 17:51:48.293617 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:48.293590 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-84.ec2.internal\" not found" Oct 30 17:51:48.394172 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:48.394118 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-84.ec2.internal\" not found" Oct 30 17:51:48.416444 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:48.416406 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 17:46:47 +0000 UTC" deadline="2028-06-03 12:08:04.292208194 +0000 UTC" Oct 30 17:51:48.416589 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:48.416447 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13962h16m15.875766681s" Oct 30 17:51:48.495043 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:48.495013 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-84.ec2.internal\" not found" Oct 30 17:51:48.518686 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:48.518653 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda45dfd37aee5c345465266021945ad3.slice/crio-90dd88d0b6a648f009bdfc07e21ba45f5aafa9255dfd19e15012a0509f9d72a0 WatchSource:0}: Error finding container 90dd88d0b6a648f009bdfc07e21ba45f5aafa9255dfd19e15012a0509f9d72a0: Status 404 returned error can't find the container with id 90dd88d0b6a648f009bdfc07e21ba45f5aafa9255dfd19e15012a0509f9d72a0 Oct 30 17:51:48.519251 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:48.519229 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ed893d4ed91b2f423e686d98f2bd57e.slice/crio-b286dc59e270f5f38fd27ef90603fbbd38a397f5db1f21bfa70e2e9caa3fe25e WatchSource:0}: Error finding container b286dc59e270f5f38fd27ef90603fbbd38a397f5db1f21bfa70e2e9caa3fe25e: Status 404 returned error can't find the container with id b286dc59e270f5f38fd27ef90603fbbd38a397f5db1f21bfa70e2e9caa3fe25e Oct 30 17:51:48.523550 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:48.523534 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:51:48.544435 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:48.544412 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:51:48.545360 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:48.545344 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-84.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:51:48.545433 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:48.545375 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-84.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:51:48.545433 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:48.545386 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-84.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:51:48.595413 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:48.595374 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-84.ec2.internal\" not found" Oct 30 17:51:48.695972 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:48.695890 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-84.ec2.internal\" not found" Oct 30 17:51:48.796563 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:48.796528 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-84.ec2.internal\" not found" Oct 30 17:51:48.837171 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:48.837141 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:51:48.871075 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:48.871044 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:51:48.897177 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:48.897131 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-84.ec2.internal\" not found" Oct 30 17:51:48.942067 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:48.942030 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:51:48.955530 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:48.955481 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal" Oct 30 17:51:48.985039 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:48.983817 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:51:48.985252 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:48.985148 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-84.ec2.internal" Oct 30 17:51:49.007448 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.007408 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:51:49.330035 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.329999 2579 apiserver.go:52] "Watching apiserver" Oct 30 17:51:49.357114 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.357082 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 17:51:49.362069 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.362037 2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 17:51:49.362459 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.362432 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal","openshift-network-diagnostics/network-check-target-jgtj4","kube-system/konnectivity-agent-qzhll","openshift-ovn-kubernetes/ovnkube-node-b4dtp","kube-system/kube-apiserver-proxy-ip-10-0-130-84.ec2.internal","openshift-image-registry/node-ca-k75vq","openshift-multus/multus-additional-cni-plugins-4bkwl","openshift-multus/network-metrics-daemon-stxf2","openshift-cluster-node-tuning-operator/tuned-t7knk","openshift-dns/node-resolver-bs5gk","openshift-multus/multus-sgp8n","openshift-network-operator/iptables-alerter-6826k","kube-system/global-pull-secret-syncer-pxv85"] Oct 30 17:51:49.364110 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.364083 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k75vq" Oct 30 17:51:49.365321 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.365293 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6826k" Oct 30 17:51:49.366543 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.366520 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgtj4" Oct 30 17:51:49.366642 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:49.366617 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgtj4" podUID="cb68da46-11b6-48f4-b302-3ad63d93f82f" Oct 30 17:51:49.367801 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.367783 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qzhll" Oct 30 17:51:49.369241 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.369219 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.370432 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.370398 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:51:49.370535 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:49.370489 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxv85" podUID="1fb157ba-42c2-4efc-a76b-6863fc987fb6" Oct 30 17:51:49.372040 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.372018 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stxf2" Oct 30 17:51:49.372142 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:49.372076 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stxf2" podUID="13521cae-dc8a-4670-9321-61ad6ab3fd4d" Oct 30 17:51:49.373189 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.373169 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:51:49.375034 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.375013 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:51:49.376739 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.376718 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.378639 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.378618 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bs5gk" Oct 30 17:51:49.380444 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.380424 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.384601 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.384581 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-twdt2\"" Oct 30 17:51:49.384943 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.384927 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 17:51:49.385032 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.384982 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 17:51:49.385032 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.384989 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 17:51:49.385032 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.384929 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:51:49.387328 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.387306 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 17:51:49.387449 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.387306 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 17:51:49.387449 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.387353 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 17:51:49.388107 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.388026 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hhs6k\"" Oct 30 17:51:49.388107 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.388076 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-k5mms\"" Oct 30 17:51:49.388484 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.388461 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 17:51:49.390390 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.390372 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 17:51:49.390509 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.390489 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 17:51:49.390671 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.390652 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fzft4\"" Oct 30 17:51:49.420523 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.420438 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 17:51:49.420523 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.420519 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qnvx2" Oct 30 17:51:49.458042 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.458013 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 17:51:49.467253 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.467223 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75cac6ab-2f87-4813-80a1-cc6efb29d187-host\") pod \"node-ca-k75vq\" (UID: \"75cac6ab-2f87-4813-80a1-cc6efb29d187\") " pod="openshift-image-registry/node-ca-k75vq" Oct 30 17:51:49.467432 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.467265 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cbda8c42-0f24-41a3-8a5f-727a4094d021-cni-binary-copy\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:51:49.467432 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.467296 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cbda8c42-0f24-41a3-8a5f-727a4094d021-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:51:49.467432 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.467387 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-slash\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.467618 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.467440 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-run-netns\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.467618 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.467485 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f983c48d-08c4-41e2-af03-7c36cf651faa-ovnkube-script-lib\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.467618 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.467544 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vbfw\" (UniqueName: \"kubernetes.io/projected/13521cae-dc8a-4670-9321-61ad6ab3fd4d-kube-api-access-5vbfw\") pod \"network-metrics-daemon-stxf2\" (UID: \"13521cae-dc8a-4670-9321-61ad6ab3fd4d\") " pod="openshift-multus/network-metrics-daemon-stxf2" Oct 30 17:51:49.467618 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.467595 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c126d55e-cb37-4070-904d-47f76d781751-tmp\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.467768 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.467664 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f348173-6e5e-404a-9126-7aa648f7eb01-host-slash\") pod \"iptables-alerter-6826k\" (UID: \"8f348173-6e5e-404a-9126-7aa648f7eb01\") " pod="openshift-network-operator/iptables-alerter-6826k" Oct 30 17:51:49.467768 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.467721 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cbda8c42-0f24-41a3-8a5f-727a4094d021-cnibin\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:51:49.467768 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.467755 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-systemd-units\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.467904 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.467783 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-registration-dir\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:51:49.467904 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.467820 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-device-dir\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:51:49.467904 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.467847 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbw76\" (UniqueName: \"kubernetes.io/projected/c126d55e-cb37-4070-904d-47f76d781751-kube-api-access-sbw76\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.467904 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.467869 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-run-multus-certs\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.467904 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.467886 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-etc-kubernetes\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.467904 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.467900 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1fb157ba-42c2-4efc-a76b-6863fc987fb6-dbus\") pod \"global-pull-secret-syncer-pxv85\" (UID: \"1fb157ba-42c2-4efc-a76b-6863fc987fb6\") " pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:51:49.468208 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.467919 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cbda8c42-0f24-41a3-8a5f-727a4094d021-system-cni-dir\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:51:49.468208 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.467976 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cbda8c42-0f24-41a3-8a5f-727a4094d021-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:51:49.468208 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.468001 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-etc-openvswitch\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.468208 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.468026 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljxjz\" (UniqueName: \"kubernetes.io/projected/f983c48d-08c4-41e2-af03-7c36cf651faa-kube-api-access-ljxjz\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.468208 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.468045 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l864g\" (UniqueName: \"kubernetes.io/projected/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-kube-api-access-l864g\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:51:49.468208 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.468073 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-sysconfig\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.468208 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.468108 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8f348173-6e5e-404a-9126-7aa648f7eb01-iptables-alerter-script\") pod \"iptables-alerter-6826k\" (UID: \"8f348173-6e5e-404a-9126-7aa648f7eb01\") " pod="openshift-network-operator/iptables-alerter-6826k" Oct 30 17:51:49.468208 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.468129 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1397e4e-e615-4954-8486-af559b65687e-tmp-dir\") pod \"node-resolver-bs5gk\" (UID: \"e1397e4e-e615-4954-8486-af559b65687e\") " pod="openshift-dns/node-resolver-bs5gk" Oct 30 17:51:49.468208 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.468161 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-kubelet\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.468208 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.468183 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-etc-selinux\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:51:49.468678 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.468274 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-sysctl-d\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.468678 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.468298 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-host\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.468678 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.468315 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/73dbe36c-9ccb-47d6-b11e-afc35b87008e-agent-certs\") pod \"konnectivity-agent-qzhll\" (UID: \"73dbe36c-9ccb-47d6-b11e-afc35b87008e\") " pod="kube-system/konnectivity-agent-qzhll" Oct 30 17:51:49.468678 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.468337 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/73dbe36c-9ccb-47d6-b11e-afc35b87008e-konnectivity-ca\") pod \"konnectivity-agent-qzhll\" (UID: \"73dbe36c-9ccb-47d6-b11e-afc35b87008e\") " pod="kube-system/konnectivity-agent-qzhll" Oct 30 17:51:49.468678 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.468363 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1fb157ba-42c2-4efc-a76b-6863fc987fb6-kubelet-config\") pod \"global-pull-secret-syncer-pxv85\" (UID: \"1fb157ba-42c2-4efc-a76b-6863fc987fb6\") " pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:51:49.468678 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.468392 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-run-openvswitch\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.468678 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.468429 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-node-log\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.468678 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.468507 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.468678 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.468559 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f983c48d-08c4-41e2-af03-7c36cf651faa-env-overrides\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.468678 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.468611 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f983c48d-08c4-41e2-af03-7c36cf651faa-ovn-node-metrics-cert\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.468678 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.468645 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbjzl\" (UniqueName: \"kubernetes.io/projected/75cac6ab-2f87-4813-80a1-cc6efb29d187-kube-api-access-pbjzl\") pod \"node-ca-k75vq\" (UID: \"75cac6ab-2f87-4813-80a1-cc6efb29d187\") " pod="openshift-image-registry/node-ca-k75vq" Oct 30 17:51:49.469122 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.468680 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f983c48d-08c4-41e2-af03-7c36cf651faa-ovnkube-config\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.469122 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.468793 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-socket-dir\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:51:49.469122 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.468858 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-sys\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.469122 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.468883 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-multus-cni-dir\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.469122 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.468921 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-run-k8s-cni-cncf-io\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.469122 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.468955 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccflp\" (UniqueName: \"kubernetes.io/projected/cbda8c42-0f24-41a3-8a5f-727a4094d021-kube-api-access-ccflp\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:51:49.469122 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469004 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2d2s\" (UniqueName: \"kubernetes.io/projected/e1397e4e-e615-4954-8486-af559b65687e-kube-api-access-b2d2s\") pod \"node-resolver-bs5gk\" (UID: \"e1397e4e-e615-4954-8486-af559b65687e\") " pod="openshift-dns/node-resolver-bs5gk" Oct 30 17:51:49.469122 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469035 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-sys-fs\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:51:49.469122 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469068 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-kubernetes\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.469122 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469108 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-cnibin\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.469531 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469148 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-var-lib-cni-bin\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.469531 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469191 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/75cac6ab-2f87-4813-80a1-cc6efb29d187-serviceca\") pod \"node-ca-k75vq\" (UID: \"75cac6ab-2f87-4813-80a1-cc6efb29d187\") " pod="openshift-image-registry/node-ca-k75vq" Oct 30 17:51:49.469531 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469232 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rhhq\" (UniqueName: \"kubernetes.io/projected/8f348173-6e5e-404a-9126-7aa648f7eb01-kube-api-access-5rhhq\") pod \"iptables-alerter-6826k\" (UID: \"8f348173-6e5e-404a-9126-7aa648f7eb01\") " pod="openshift-network-operator/iptables-alerter-6826k" Oct 30 17:51:49.469531 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469294 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cbda8c42-0f24-41a3-8a5f-727a4094d021-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:51:49.469531 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469321 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-run-netns\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.469531 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469385 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q25cz\" (UniqueName: \"kubernetes.io/projected/63b2e2e0-805f-4197-81d5-6eafe3831ded-kube-api-access-q25cz\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.469531 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469404 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-modprobe-d\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.469531 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469423 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-var-lib-cni-multus\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.469531 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469449 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1fb157ba-42c2-4efc-a76b-6863fc987fb6-original-pull-secret\") pod \"global-pull-secret-syncer-pxv85\" (UID: \"1fb157ba-42c2-4efc-a76b-6863fc987fb6\") " pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:51:49.469531 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469490 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-run-ovn\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.469531 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469505 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-cni-bin\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.469531 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469520 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-system-cni-dir\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.469996 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469545 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-multus-socket-dir-parent\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.469996 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469566 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-multus-conf-dir\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.469996 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469592 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-run-systemd\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.469996 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469620 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-var-lib-openvswitch\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.469996 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469650 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/63b2e2e0-805f-4197-81d5-6eafe3831ded-cni-binary-copy\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.469996 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469720 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-hostroot\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.469996 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469739 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e1397e4e-e615-4954-8486-af559b65687e-hosts-file\") pod \"node-resolver-bs5gk\" (UID: \"e1397e4e-e615-4954-8486-af559b65687e\") " pod="openshift-dns/node-resolver-bs5gk" Oct 30 17:51:49.469996 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469763 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-cni-netd\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.469996 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469788 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13521cae-dc8a-4670-9321-61ad6ab3fd4d-metrics-certs\") pod \"network-metrics-daemon-stxf2\" (UID: \"13521cae-dc8a-4670-9321-61ad6ab3fd4d\") " pod="openshift-multus/network-metrics-daemon-stxf2" Oct 30 17:51:49.469996 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469820 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:51:49.469996 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469837 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-lib-modules\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.469996 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469857 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/63b2e2e0-805f-4197-81d5-6eafe3831ded-multus-daemon-config\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.469996 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469871 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-log-socket\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.469996 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469886 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-var-lib-kubelet\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.469996 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469902 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-var-lib-kubelet\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.469996 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469917 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mppvt\" (UniqueName: \"kubernetes.io/projected/cb68da46-11b6-48f4-b302-3ad63d93f82f-kube-api-access-mppvt\") pod \"network-check-target-jgtj4\" (UID: \"cb68da46-11b6-48f4-b302-3ad63d93f82f\") " pod="openshift-network-diagnostics/network-check-target-jgtj4" Oct 30 17:51:49.470686 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469933 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-run-ovn-kubernetes\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.470686 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469948 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-sysctl-conf\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.470686 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469962 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-systemd\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.470686 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469978 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-run\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.470686 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.469998 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c126d55e-cb37-4070-904d-47f76d781751-etc-tuned\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.498179 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.498149 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 17:51:49.498379 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.498239 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 17:51:49.498379 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.498274 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 17:51:49.498379 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.498320 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:51:49.498510 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.498383 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 17:51:49.498510 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.498402 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 17:51:49.498510 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.498413 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 17:51:49.498510 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.498417 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-dqbz4\"" Oct 30 17:51:49.498510 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.498454 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 17:51:49.498510 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.498462 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lsd9l\"" Oct 30 17:51:49.498510 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.498471 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 17:51:49.498510 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.498499 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 17:51:49.498754 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.498515 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 17:51:49.498754 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.498520 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5kfdv\"" Oct 30 17:51:49.498754 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.498525 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 17:51:49.498754 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.498465 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nb8hn\"" Oct 30 17:51:49.498754 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.498545 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 17:51:49.498754 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.498569 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 17:51:49.498754 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.498579 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 17:51:49.498754 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.498587 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-k56gt\"" Oct 30 17:51:49.498754 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.498598 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 17:51:49.498754 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.498616 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 17:51:49.503912 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.503844 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-84.ec2.internal" event={"ID":"da45dfd37aee5c345465266021945ad3","Type":"ContainerStarted","Data":"90dd88d0b6a648f009bdfc07e21ba45f5aafa9255dfd19e15012a0509f9d72a0"} Oct 30 17:51:49.505497 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.505467 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal" event={"ID":"5ed893d4ed91b2f423e686d98f2bd57e","Type":"ContainerStarted","Data":"b286dc59e270f5f38fd27ef90603fbbd38a397f5db1f21bfa70e2e9caa3fe25e"} Oct 30 17:51:49.571074 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.571045 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-sys\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.571276 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.571081 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-multus-cni-dir\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.571276 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.571107 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-run-k8s-cni-cncf-io\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.571276 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.571131 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccflp\" (UniqueName: \"kubernetes.io/projected/cbda8c42-0f24-41a3-8a5f-727a4094d021-kube-api-access-ccflp\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:51:49.571276 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.571154 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2d2s\" (UniqueName: \"kubernetes.io/projected/e1397e4e-e615-4954-8486-af559b65687e-kube-api-access-b2d2s\") pod \"node-resolver-bs5gk\" (UID: \"e1397e4e-e615-4954-8486-af559b65687e\") " pod="openshift-dns/node-resolver-bs5gk" Oct 30 17:51:49.571276 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.571177 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-sys-fs\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:51:49.571276 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.571201 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-kubernetes\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.571276 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.571223 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-cnibin\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.571276 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.571251 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-var-lib-cni-bin\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.571276 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.571277 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/75cac6ab-2f87-4813-80a1-cc6efb29d187-serviceca\") pod \"node-ca-k75vq\" (UID: \"75cac6ab-2f87-4813-80a1-cc6efb29d187\") " pod="openshift-image-registry/node-ca-k75vq" Oct 30 17:51:49.571781 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.571301 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rhhq\" (UniqueName: \"kubernetes.io/projected/8f348173-6e5e-404a-9126-7aa648f7eb01-kube-api-access-5rhhq\") pod \"iptables-alerter-6826k\" (UID: \"8f348173-6e5e-404a-9126-7aa648f7eb01\") " pod="openshift-network-operator/iptables-alerter-6826k" Oct 30 17:51:49.571781 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.571332 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cbda8c42-0f24-41a3-8a5f-727a4094d021-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:51:49.571781 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.571379 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-run-netns\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.571781 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.571405 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q25cz\" (UniqueName: \"kubernetes.io/projected/63b2e2e0-805f-4197-81d5-6eafe3831ded-kube-api-access-q25cz\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.571781 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.571430 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-modprobe-d\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.571781 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.571457 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-var-lib-cni-multus\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.571781 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.571483 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1fb157ba-42c2-4efc-a76b-6863fc987fb6-original-pull-secret\") pod \"global-pull-secret-syncer-pxv85\" (UID: \"1fb157ba-42c2-4efc-a76b-6863fc987fb6\") " pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:51:49.571781 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.571521 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-run-ovn\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.571781 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.571551 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-cni-bin\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.571781 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.571576 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-system-cni-dir\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.571781 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.571602 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-multus-socket-dir-parent\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.571781 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.571626 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-multus-conf-dir\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.571781 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.571651 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-run-systemd\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.571781 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.571676 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-var-lib-openvswitch\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.571781 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.571700 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/63b2e2e0-805f-4197-81d5-6eafe3831ded-cni-binary-copy\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.571781 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.571726 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-hostroot\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.571781 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.571753 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e1397e4e-e615-4954-8486-af559b65687e-hosts-file\") pod \"node-resolver-bs5gk\" (UID: \"e1397e4e-e615-4954-8486-af559b65687e\") " pod="openshift-dns/node-resolver-bs5gk" Oct 30 17:51:49.572679 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.571793 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-cni-netd\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.572679 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.571827 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13521cae-dc8a-4670-9321-61ad6ab3fd4d-metrics-certs\") pod \"network-metrics-daemon-stxf2\" (UID: \"13521cae-dc8a-4670-9321-61ad6ab3fd4d\") " pod="openshift-multus/network-metrics-daemon-stxf2" Oct 30 17:51:49.572679 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.571852 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:51:49.572679 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.571876 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-lib-modules\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.572679 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.571902 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/63b2e2e0-805f-4197-81d5-6eafe3831ded-multus-daemon-config\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.572679 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.571926 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-log-socket\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.572679 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.571952 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-var-lib-kubelet\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.572679 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.571979 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-var-lib-kubelet\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.572679 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.572005 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mppvt\" (UniqueName: \"kubernetes.io/projected/cb68da46-11b6-48f4-b302-3ad63d93f82f-kube-api-access-mppvt\") pod \"network-check-target-jgtj4\" (UID: \"cb68da46-11b6-48f4-b302-3ad63d93f82f\") " pod="openshift-network-diagnostics/network-check-target-jgtj4" Oct 30 17:51:49.572679 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.572036 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-run-ovn-kubernetes\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.572679 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.572100 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-sysctl-conf\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.572679 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.572125 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-systemd\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.572679 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.572150 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-run\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.572679 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.572190 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c126d55e-cb37-4070-904d-47f76d781751-etc-tuned\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.572679 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.572214 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75cac6ab-2f87-4813-80a1-cc6efb29d187-host\") pod \"node-ca-k75vq\" (UID: \"75cac6ab-2f87-4813-80a1-cc6efb29d187\") " pod="openshift-image-registry/node-ca-k75vq" Oct 30 17:51:49.572679 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.572293 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cbda8c42-0f24-41a3-8a5f-727a4094d021-cni-binary-copy\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:51:49.572679 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.572342 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cbda8c42-0f24-41a3-8a5f-727a4094d021-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:51:49.573562 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.572371 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-slash\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.573562 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.572425 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-run-netns\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.573562 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.572451 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f983c48d-08c4-41e2-af03-7c36cf651faa-ovnkube-script-lib\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.573562 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.572484 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vbfw\" (UniqueName: \"kubernetes.io/projected/13521cae-dc8a-4670-9321-61ad6ab3fd4d-kube-api-access-5vbfw\") pod \"network-metrics-daemon-stxf2\" (UID: \"13521cae-dc8a-4670-9321-61ad6ab3fd4d\") " pod="openshift-multus/network-metrics-daemon-stxf2" Oct 30 17:51:49.573562 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.572510 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c126d55e-cb37-4070-904d-47f76d781751-tmp\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.573562 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.572536 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f348173-6e5e-404a-9126-7aa648f7eb01-host-slash\") pod \"iptables-alerter-6826k\" (UID: \"8f348173-6e5e-404a-9126-7aa648f7eb01\") " pod="openshift-network-operator/iptables-alerter-6826k" Oct 30 17:51:49.573562 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.572573 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cbda8c42-0f24-41a3-8a5f-727a4094d021-cnibin\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:51:49.573562 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.572605 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-systemd-units\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.573562 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.572631 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-registration-dir\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:51:49.573562 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.572663 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-device-dir\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:51:49.573562 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.572698 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbw76\" (UniqueName: \"kubernetes.io/projected/c126d55e-cb37-4070-904d-47f76d781751-kube-api-access-sbw76\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.573562 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.572725 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-run-multus-certs\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.573562 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.572758 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-etc-kubernetes\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.573562 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.572790 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1fb157ba-42c2-4efc-a76b-6863fc987fb6-dbus\") pod \"global-pull-secret-syncer-pxv85\" (UID: \"1fb157ba-42c2-4efc-a76b-6863fc987fb6\") " pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:51:49.573562 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.572852 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cbda8c42-0f24-41a3-8a5f-727a4094d021-system-cni-dir\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:51:49.573562 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.572880 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cbda8c42-0f24-41a3-8a5f-727a4094d021-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:51:49.573562 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.572907 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-etc-openvswitch\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.574322 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.572948 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljxjz\" (UniqueName: \"kubernetes.io/projected/f983c48d-08c4-41e2-af03-7c36cf651faa-kube-api-access-ljxjz\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.574322 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.572981 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l864g\" (UniqueName: \"kubernetes.io/projected/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-kube-api-access-l864g\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:51:49.574322 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.573009 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-sysconfig\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.574322 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.573039 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8f348173-6e5e-404a-9126-7aa648f7eb01-iptables-alerter-script\") pod \"iptables-alerter-6826k\" (UID: \"8f348173-6e5e-404a-9126-7aa648f7eb01\") " pod="openshift-network-operator/iptables-alerter-6826k" Oct 30 17:51:49.574322 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.573081 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1397e4e-e615-4954-8486-af559b65687e-tmp-dir\") pod \"node-resolver-bs5gk\" (UID: \"e1397e4e-e615-4954-8486-af559b65687e\") " pod="openshift-dns/node-resolver-bs5gk" Oct 30 17:51:49.574322 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.573110 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-kubelet\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.574322 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.573140 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-etc-selinux\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:51:49.574322 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.573186 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-sysctl-d\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.574322 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.573213 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-host\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.574322 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.573252 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/73dbe36c-9ccb-47d6-b11e-afc35b87008e-agent-certs\") pod \"konnectivity-agent-qzhll\" (UID: \"73dbe36c-9ccb-47d6-b11e-afc35b87008e\") " pod="kube-system/konnectivity-agent-qzhll" Oct 30 17:51:49.574322 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.573284 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/73dbe36c-9ccb-47d6-b11e-afc35b87008e-konnectivity-ca\") pod \"konnectivity-agent-qzhll\" (UID: \"73dbe36c-9ccb-47d6-b11e-afc35b87008e\") " pod="kube-system/konnectivity-agent-qzhll" Oct 30 17:51:49.574322 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.573316 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1fb157ba-42c2-4efc-a76b-6863fc987fb6-kubelet-config\") pod \"global-pull-secret-syncer-pxv85\" (UID: \"1fb157ba-42c2-4efc-a76b-6863fc987fb6\") " pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:51:49.574322 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.573344 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-run-openvswitch\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.574322 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.573425 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-node-log\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.574322 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.573467 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.574322 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.573496 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f983c48d-08c4-41e2-af03-7c36cf651faa-env-overrides\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.574322 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.573522 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f983c48d-08c4-41e2-af03-7c36cf651faa-ovn-node-metrics-cert\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.575240 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.573555 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbjzl\" (UniqueName: \"kubernetes.io/projected/75cac6ab-2f87-4813-80a1-cc6efb29d187-kube-api-access-pbjzl\") pod \"node-ca-k75vq\" (UID: \"75cac6ab-2f87-4813-80a1-cc6efb29d187\") " pod="openshift-image-registry/node-ca-k75vq" Oct 30 17:51:49.575240 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.573593 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f983c48d-08c4-41e2-af03-7c36cf651faa-ovnkube-config\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.575240 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.573631 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-socket-dir\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:51:49.575240 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.573869 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-socket-dir\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:51:49.575240 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.573889 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-sys\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.575240 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.574107 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-multus-cni-dir\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.575240 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.574179 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75cac6ab-2f87-4813-80a1-cc6efb29d187-host\") pod \"node-ca-k75vq\" (UID: \"75cac6ab-2f87-4813-80a1-cc6efb29d187\") " pod="openshift-image-registry/node-ca-k75vq" Oct 30 17:51:49.575240 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.574279 2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 17:51:49.575240 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.574881 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cbda8c42-0f24-41a3-8a5f-727a4094d021-cni-binary-copy\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:51:49.575240 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.575095 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cbda8c42-0f24-41a3-8a5f-727a4094d021-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:51:49.575240 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.575156 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-slash\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.575240 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.575203 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-run-netns\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.575871 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.575759 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-sysconfig\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.575871 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.575795 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f983c48d-08c4-41e2-af03-7c36cf651faa-ovnkube-script-lib\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.576057 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.576028 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-run-k8s-cni-cncf-io\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.576350 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.576313 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8f348173-6e5e-404a-9126-7aa648f7eb01-iptables-alerter-script\") pod \"iptables-alerter-6826k\" (UID: \"8f348173-6e5e-404a-9126-7aa648f7eb01\") " pod="openshift-network-operator/iptables-alerter-6826k" Oct 30 17:51:49.576572 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.576546 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-sys-fs\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:51:49.576666 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.576609 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-kubernetes\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.576666 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.576634 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1397e4e-e615-4954-8486-af559b65687e-tmp-dir\") pod \"node-resolver-bs5gk\" (UID: \"e1397e4e-e615-4954-8486-af559b65687e\") " pod="openshift-dns/node-resolver-bs5gk" Oct 30 17:51:49.576666 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.576660 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-cnibin\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.576824 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.576709 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-var-lib-cni-bin\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.576824 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.576720 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-kubelet\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.576921 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.576832 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-etc-selinux\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:51:49.576965 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.576946 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-sysctl-d\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.577015 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.576999 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-host\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.577198 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.577174 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-run-multus-certs\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.577198 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.577192 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-systemd-units\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.577413 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.577259 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-registration-dir\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:51:49.577413 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.577322 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f348173-6e5e-404a-9126-7aa648f7eb01-host-slash\") pod \"iptables-alerter-6826k\" (UID: \"8f348173-6e5e-404a-9126-7aa648f7eb01\") " pod="openshift-network-operator/iptables-alerter-6826k" Oct 30 17:51:49.577413 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.577297 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-device-dir\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:51:49.578542 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.577680 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-hostroot\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.578542 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.577736 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/75cac6ab-2f87-4813-80a1-cc6efb29d187-serviceca\") pod \"node-ca-k75vq\" (UID: \"75cac6ab-2f87-4813-80a1-cc6efb29d187\") " pod="openshift-image-registry/node-ca-k75vq" Oct 30 17:51:49.578542 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.577751 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1fb157ba-42c2-4efc-a76b-6863fc987fb6-kubelet-config\") pod \"global-pull-secret-syncer-pxv85\" (UID: \"1fb157ba-42c2-4efc-a76b-6863fc987fb6\") " pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:51:49.578542 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.577770 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-etc-openvswitch\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.578542 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.577755 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-run-netns\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.578542 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.577808 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-run-openvswitch\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.578542 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.577813 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e1397e4e-e615-4954-8486-af559b65687e-hosts-file\") pod \"node-resolver-bs5gk\" (UID: \"e1397e4e-e615-4954-8486-af559b65687e\") " pod="openshift-dns/node-resolver-bs5gk" Oct 30 17:51:49.578542 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.577861 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-cni-netd\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.578542 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.577900 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cbda8c42-0f24-41a3-8a5f-727a4094d021-cnibin\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:51:49.578542 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.577920 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-node-log\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.578542 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:49.577952 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:51:49.578542 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.577959 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.578542 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:49.578035 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13521cae-dc8a-4670-9321-61ad6ab3fd4d-metrics-certs podName:13521cae-dc8a-4670-9321-61ad6ab3fd4d nodeName:}" failed. No retries permitted until 2026-10-30 17:51:50.078010521 +0000 UTC m=+3.096967693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13521cae-dc8a-4670-9321-61ad6ab3fd4d-metrics-certs") pod "network-metrics-daemon-stxf2" (UID: "13521cae-dc8a-4670-9321-61ad6ab3fd4d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:51:49.578542 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.578094 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:51:49.578542 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.578098 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-modprobe-d\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.578542 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.578094 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-var-lib-cni-multus\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.578542 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.578148 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-cni-bin\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.578542 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:49.578188 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:51:49.579504 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.578209 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-run-systemd\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.579504 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.578231 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-lib-modules\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.579504 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:49.578257 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fb157ba-42c2-4efc-a76b-6863fc987fb6-original-pull-secret podName:1fb157ba-42c2-4efc-a76b-6863fc987fb6 nodeName:}" failed. No retries permitted until 2026-10-30 17:51:50.07823836 +0000 UTC m=+3.097195533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1fb157ba-42c2-4efc-a76b-6863fc987fb6-original-pull-secret") pod "global-pull-secret-syncer-pxv85" (UID: "1fb157ba-42c2-4efc-a76b-6863fc987fb6") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:51:49.579504 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.578269 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-multus-socket-dir-parent\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.579504 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.578297 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1fb157ba-42c2-4efc-a76b-6863fc987fb6-dbus\") pod \"global-pull-secret-syncer-pxv85\" (UID: \"1fb157ba-42c2-4efc-a76b-6863fc987fb6\") " pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:51:49.579504 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.578516 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-multus-conf-dir\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.579504 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.578558 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-etc-kubernetes\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.579504 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.578570 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f983c48d-08c4-41e2-af03-7c36cf651faa-env-overrides\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.579504 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.578620 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-run-ovn-kubernetes\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.579504 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.578644 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-system-cni-dir\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.579504 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.578672 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-log-socket\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.579504 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.578703 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c126d55e-cb37-4070-904d-47f76d781751-etc-tuned\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.579504 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.578740 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-var-lib-kubelet\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.579504 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.578762 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-run-ovn\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.579504 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.578790 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-var-lib-kubelet\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.579504 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.578823 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-systemd\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.579504 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.578838 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/63b2e2e0-805f-4197-81d5-6eafe3831ded-multus-daemon-config\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.579992 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.578881 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-var-lib-openvswitch\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.579992 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.578925 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-run\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.579992 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.578945 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-sysctl-conf\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.579992 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.578999 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cbda8c42-0f24-41a3-8a5f-727a4094d021-system-cni-dir\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:51:49.579992 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.579159 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f983c48d-08c4-41e2-af03-7c36cf651faa-ovnkube-config\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.579992 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.579508 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cbda8c42-0f24-41a3-8a5f-727a4094d021-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:51:49.579992 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.579532 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/63b2e2e0-805f-4197-81d5-6eafe3831ded-cni-binary-copy\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.579992 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.579693 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cbda8c42-0f24-41a3-8a5f-727a4094d021-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:51:49.579992 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.579792 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/73dbe36c-9ccb-47d6-b11e-afc35b87008e-konnectivity-ca\") pod \"konnectivity-agent-qzhll\" (UID: \"73dbe36c-9ccb-47d6-b11e-afc35b87008e\") " pod="kube-system/konnectivity-agent-qzhll" Oct 30 17:51:49.581843 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.581762 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c126d55e-cb37-4070-904d-47f76d781751-tmp\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.582547 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.582521 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f983c48d-08c4-41e2-af03-7c36cf651faa-ovn-node-metrics-cert\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.582631 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.582572 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/73dbe36c-9ccb-47d6-b11e-afc35b87008e-agent-certs\") pod \"konnectivity-agent-qzhll\" (UID: \"73dbe36c-9ccb-47d6-b11e-afc35b87008e\") " pod="kube-system/konnectivity-agent-qzhll" Oct 30 17:51:49.602713 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.602683 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccflp\" (UniqueName: \"kubernetes.io/projected/cbda8c42-0f24-41a3-8a5f-727a4094d021-kube-api-access-ccflp\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:51:49.603980 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:49.603955 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:51:49.604073 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:49.603985 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:51:49.604073 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:49.604000 2579 projected.go:196] Error preparing data for projected volume kube-api-access-mppvt for pod openshift-network-diagnostics/network-check-target-jgtj4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:51:49.604195 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:49.604075 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb68da46-11b6-48f4-b302-3ad63d93f82f-kube-api-access-mppvt podName:cb68da46-11b6-48f4-b302-3ad63d93f82f nodeName:}" failed. No retries permitted until 2026-10-30 17:51:50.104054371 +0000 UTC m=+3.123011545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mppvt" (UniqueName: "kubernetes.io/projected/cb68da46-11b6-48f4-b302-3ad63d93f82f-kube-api-access-mppvt") pod "network-check-target-jgtj4" (UID: "cb68da46-11b6-48f4-b302-3ad63d93f82f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:51:49.608917 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.608251 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbw76\" (UniqueName: \"kubernetes.io/projected/c126d55e-cb37-4070-904d-47f76d781751-kube-api-access-sbw76\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.608917 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.608697 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbjzl\" (UniqueName: \"kubernetes.io/projected/75cac6ab-2f87-4813-80a1-cc6efb29d187-kube-api-access-pbjzl\") pod \"node-ca-k75vq\" (UID: \"75cac6ab-2f87-4813-80a1-cc6efb29d187\") " pod="openshift-image-registry/node-ca-k75vq" Oct 30 17:51:49.608917 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.608737 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l864g\" (UniqueName: \"kubernetes.io/projected/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-kube-api-access-l864g\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:51:49.612037 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.609708 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljxjz\" (UniqueName: \"kubernetes.io/projected/f983c48d-08c4-41e2-af03-7c36cf651faa-kube-api-access-ljxjz\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.612037 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.610571 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q25cz\" (UniqueName: \"kubernetes.io/projected/63b2e2e0-805f-4197-81d5-6eafe3831ded-kube-api-access-q25cz\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.612037 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.610924 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2d2s\" (UniqueName: \"kubernetes.io/projected/e1397e4e-e615-4954-8486-af559b65687e-kube-api-access-b2d2s\") pod \"node-resolver-bs5gk\" (UID: \"e1397e4e-e615-4954-8486-af559b65687e\") " pod="openshift-dns/node-resolver-bs5gk" Oct 30 17:51:49.612680 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.612654 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rhhq\" (UniqueName: \"kubernetes.io/projected/8f348173-6e5e-404a-9126-7aa648f7eb01-kube-api-access-5rhhq\") pod \"iptables-alerter-6826k\" (UID: \"8f348173-6e5e-404a-9126-7aa648f7eb01\") " pod="openshift-network-operator/iptables-alerter-6826k" Oct 30 17:51:49.613003 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.612954 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vbfw\" (UniqueName: \"kubernetes.io/projected/13521cae-dc8a-4670-9321-61ad6ab3fd4d-kube-api-access-5vbfw\") pod \"network-metrics-daemon-stxf2\" (UID: \"13521cae-dc8a-4670-9321-61ad6ab3fd4d\") " pod="openshift-multus/network-metrics-daemon-stxf2" Oct 30 17:51:49.678299 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.678270 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k75vq" Oct 30 17:51:49.688701 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.688678 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6826k" Oct 30 17:51:49.689757 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:49.689723 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75cac6ab_2f87_4813_80a1_cc6efb29d187.slice/crio-a3663bec9c925c88b65e04235931d37e954e7c5375bd99f92a6eae78fe533e71 WatchSource:0}: Error finding container a3663bec9c925c88b65e04235931d37e954e7c5375bd99f92a6eae78fe533e71: Status 404 returned error can't find the container with id a3663bec9c925c88b65e04235931d37e954e7c5375bd99f92a6eae78fe533e71 Oct 30 17:51:49.698197 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:49.698164 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f348173_6e5e_404a_9126_7aa648f7eb01.slice/crio-bd497f78610265355b84c9770790898f4b6dace5069c46cbb68eebf4ba9c48ce WatchSource:0}: Error finding container bd497f78610265355b84c9770790898f4b6dace5069c46cbb68eebf4ba9c48ce: Status 404 returned error can't find the container with id bd497f78610265355b84c9770790898f4b6dace5069c46cbb68eebf4ba9c48ce Oct 30 17:51:49.701420 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.701316 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qzhll" Oct 30 17:51:49.711106 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:49.711077 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73dbe36c_9ccb_47d6_b11e_afc35b87008e.slice/crio-154701d0f49a1cff69b8eb04b125f9cdc160ef78aeafb7ac1d6a9d77708ec895 WatchSource:0}: Error finding container 154701d0f49a1cff69b8eb04b125f9cdc160ef78aeafb7ac1d6a9d77708ec895: Status 404 returned error can't find the container with id 154701d0f49a1cff69b8eb04b125f9cdc160ef78aeafb7ac1d6a9d77708ec895 Oct 30 17:51:49.724785 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.724755 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:51:49.729543 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.729509 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:51:49.737163 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.737072 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:51:49.737663 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:49.737631 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf983c48d_08c4_41e2_af03_7c36cf651faa.slice/crio-b46c2a539b3aedbbe6198927c9b223a42e6a8786a6c028393a61d883c75ae4d0 WatchSource:0}: Error finding container b46c2a539b3aedbbe6198927c9b223a42e6a8786a6c028393a61d883c75ae4d0: Status 404 returned error can't find the container with id b46c2a539b3aedbbe6198927c9b223a42e6a8786a6c028393a61d883c75ae4d0 Oct 30 17:51:49.738463 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.738441 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:51:49.740484 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:49.740453 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7db1b0_8abd_47fb_9df8_7caab8adfed6.slice/crio-cf97c2e0988205e99b5e8a6c36b3f10a7aacf72bca36e0643596739b693d573e WatchSource:0}: Error finding container cf97c2e0988205e99b5e8a6c36b3f10a7aacf72bca36e0643596739b693d573e: Status 404 returned error can't find the container with id cf97c2e0988205e99b5e8a6c36b3f10a7aacf72bca36e0643596739b693d573e Oct 30 17:51:49.741129 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.741101 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bs5gk" Oct 30 17:51:49.747460 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.747437 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sgp8n" Oct 30 17:51:49.751789 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:49.751758 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbda8c42_0f24_41a3_8a5f_727a4094d021.slice/crio-890471f8caa56baae6e88be363425b85cf5b83af0837023213d084bbcc0441ca WatchSource:0}: Error finding container 890471f8caa56baae6e88be363425b85cf5b83af0837023213d084bbcc0441ca: Status 404 returned error can't find the container with id 890471f8caa56baae6e88be363425b85cf5b83af0837023213d084bbcc0441ca Oct 30 17:51:49.759264 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:49.759233 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63b2e2e0_805f_4197_81d5_6eafe3831ded.slice/crio-3f7389d17e5cf41130e288aac5921695740d030d2937f61a40db6d65ab9f8dc6 WatchSource:0}: Error finding container 3f7389d17e5cf41130e288aac5921695740d030d2937f61a40db6d65ab9f8dc6: Status 404 returned error can't find the container with id 3f7389d17e5cf41130e288aac5921695740d030d2937f61a40db6d65ab9f8dc6 Oct 30 17:51:49.760087 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:51:49.760053 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1397e4e_e615_4954_8486_af559b65687e.slice/crio-d38f96c68a13e957832eb9e4bafbfc7db916f074f29e4a8bfd8efaf32a605148 WatchSource:0}: Error finding container d38f96c68a13e957832eb9e4bafbfc7db916f074f29e4a8bfd8efaf32a605148: Status 404 returned error can't find the container with id d38f96c68a13e957832eb9e4bafbfc7db916f074f29e4a8bfd8efaf32a605148 Oct 30 17:51:49.817715 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:49.817672 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:51:50.178709 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:50.178671 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mppvt\" (UniqueName: \"kubernetes.io/projected/cb68da46-11b6-48f4-b302-3ad63d93f82f-kube-api-access-mppvt\") pod \"network-check-target-jgtj4\" (UID: \"cb68da46-11b6-48f4-b302-3ad63d93f82f\") " pod="openshift-network-diagnostics/network-check-target-jgtj4" Oct 30 17:51:50.178871 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:50.178767 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1fb157ba-42c2-4efc-a76b-6863fc987fb6-original-pull-secret\") pod \"global-pull-secret-syncer-pxv85\" (UID: \"1fb157ba-42c2-4efc-a76b-6863fc987fb6\") " pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:51:50.178871 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:50.178805 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13521cae-dc8a-4670-9321-61ad6ab3fd4d-metrics-certs\") pod \"network-metrics-daemon-stxf2\" (UID: \"13521cae-dc8a-4670-9321-61ad6ab3fd4d\") " pod="openshift-multus/network-metrics-daemon-stxf2" Oct 30 17:51:50.179017 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:50.178929 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:51:50.179017 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:50.178986 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13521cae-dc8a-4670-9321-61ad6ab3fd4d-metrics-certs podName:13521cae-dc8a-4670-9321-61ad6ab3fd4d nodeName:}" failed. No retries permitted until 2026-10-30 17:51:51.178968621 +0000 UTC m=+4.197925793 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13521cae-dc8a-4670-9321-61ad6ab3fd4d-metrics-certs") pod "network-metrics-daemon-stxf2" (UID: "13521cae-dc8a-4670-9321-61ad6ab3fd4d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:51:50.179134 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:50.179070 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:51:50.179134 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:50.179085 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:51:50.179134 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:50.179099 2579 projected.go:196] Error preparing data for projected volume kube-api-access-mppvt for pod openshift-network-diagnostics/network-check-target-jgtj4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:51:50.179134 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:50.179134 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb68da46-11b6-48f4-b302-3ad63d93f82f-kube-api-access-mppvt podName:cb68da46-11b6-48f4-b302-3ad63d93f82f nodeName:}" failed. No retries permitted until 2026-10-30 17:51:51.179122907 +0000 UTC m=+4.198080077 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mppvt" (UniqueName: "kubernetes.io/projected/cb68da46-11b6-48f4-b302-3ad63d93f82f-kube-api-access-mppvt") pod "network-check-target-jgtj4" (UID: "cb68da46-11b6-48f4-b302-3ad63d93f82f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:51:50.179340 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:50.179189 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:51:50.179340 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:50.179239 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fb157ba-42c2-4efc-a76b-6863fc987fb6-original-pull-secret podName:1fb157ba-42c2-4efc-a76b-6863fc987fb6 nodeName:}" failed. No retries permitted until 2026-10-30 17:51:51.179213211 +0000 UTC m=+4.198170382 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1fb157ba-42c2-4efc-a76b-6863fc987fb6-original-pull-secret") pod "global-pull-secret-syncer-pxv85" (UID: "1fb157ba-42c2-4efc-a76b-6863fc987fb6") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:51:50.424041 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:50.423726 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:46:49 +0000 UTC" deadline="2028-08-10 09:15:42.9323436 +0000 UTC" Oct 30 17:51:50.424041 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:50.423951 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15591h23m52.50839528s" Oct 30 17:51:50.498648 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:50.498610 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stxf2" Oct 30 17:51:50.498839 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:50.498787 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stxf2" podUID="13521cae-dc8a-4670-9321-61ad6ab3fd4d" Oct 30 17:51:50.499281 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:50.499260 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:51:50.499423 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:50.499374 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxv85" podUID="1fb157ba-42c2-4efc-a76b-6863fc987fb6" Oct 30 17:51:50.527806 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:50.527331 2579 generic.go:356] "Generic (PLEG): container finished" podID="5ed893d4ed91b2f423e686d98f2bd57e" containerID="52fe5e042c4c2aa8ad60ef9d51dac12745b96ff0c527b5d29f7e2f139aeed1c9" exitCode=0 Oct 30 17:51:50.528049 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:50.528009 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal" event={"ID":"5ed893d4ed91b2f423e686d98f2bd57e","Type":"ContainerDied","Data":"52fe5e042c4c2aa8ad60ef9d51dac12745b96ff0c527b5d29f7e2f139aeed1c9"} Oct 30 17:51:50.533686 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:50.533655 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bs5gk" event={"ID":"e1397e4e-e615-4954-8486-af559b65687e","Type":"ContainerStarted","Data":"d38f96c68a13e957832eb9e4bafbfc7db916f074f29e4a8bfd8efaf32a605148"} Oct 30 17:51:50.544249 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:50.544146 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sgp8n" event={"ID":"63b2e2e0-805f-4197-81d5-6eafe3831ded","Type":"ContainerStarted","Data":"3f7389d17e5cf41130e288aac5921695740d030d2937f61a40db6d65ab9f8dc6"} Oct 30 17:51:50.548896 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:50.548810 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t7knk" event={"ID":"c126d55e-cb37-4070-904d-47f76d781751","Type":"ContainerStarted","Data":"53ef7808da6730a62e8044ef19d966bb7d70365a7ae4b1754b2b9ab1e69d08e1"} Oct 30 17:51:50.552571 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:50.552478 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" event={"ID":"f983c48d-08c4-41e2-af03-7c36cf651faa","Type":"ContainerStarted","Data":"b46c2a539b3aedbbe6198927c9b223a42e6a8786a6c028393a61d883c75ae4d0"} Oct 30 17:51:50.558248 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:50.557640 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-84.ec2.internal" event={"ID":"da45dfd37aee5c345465266021945ad3","Type":"ContainerStarted","Data":"5d85c45e7c121a33349d8096f9531a16efbb1c1fb886dbd0b6262e3b790d64f5"} Oct 30 17:51:50.562267 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:50.562190 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4bkwl" event={"ID":"cbda8c42-0f24-41a3-8a5f-727a4094d021","Type":"ContainerStarted","Data":"890471f8caa56baae6e88be363425b85cf5b83af0837023213d084bbcc0441ca"} Oct 30 17:51:50.564251 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:50.564170 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" event={"ID":"3b7db1b0-8abd-47fb-9df8-7caab8adfed6","Type":"ContainerStarted","Data":"cf97c2e0988205e99b5e8a6c36b3f10a7aacf72bca36e0643596739b693d573e"} Oct 30 17:51:50.579294 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:50.579263 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qzhll" event={"ID":"73dbe36c-9ccb-47d6-b11e-afc35b87008e","Type":"ContainerStarted","Data":"154701d0f49a1cff69b8eb04b125f9cdc160ef78aeafb7ac1d6a9d77708ec895"} Oct 30 17:51:50.592483 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:50.592393 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6826k" event={"ID":"8f348173-6e5e-404a-9126-7aa648f7eb01","Type":"ContainerStarted","Data":"bd497f78610265355b84c9770790898f4b6dace5069c46cbb68eebf4ba9c48ce"} Oct 30 17:51:50.601106 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:50.601073 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k75vq" event={"ID":"75cac6ab-2f87-4813-80a1-cc6efb29d187","Type":"ContainerStarted","Data":"a3663bec9c925c88b65e04235931d37e954e7c5375bd99f92a6eae78fe533e71"} Oct 30 17:51:51.190491 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:51.189547 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1fb157ba-42c2-4efc-a76b-6863fc987fb6-original-pull-secret\") pod \"global-pull-secret-syncer-pxv85\" (UID: \"1fb157ba-42c2-4efc-a76b-6863fc987fb6\") " pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:51:51.190491 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:51.189600 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13521cae-dc8a-4670-9321-61ad6ab3fd4d-metrics-certs\") pod \"network-metrics-daemon-stxf2\" (UID: \"13521cae-dc8a-4670-9321-61ad6ab3fd4d\") " pod="openshift-multus/network-metrics-daemon-stxf2" Oct 30 17:51:51.190491 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:51.189633 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mppvt\" (UniqueName: \"kubernetes.io/projected/cb68da46-11b6-48f4-b302-3ad63d93f82f-kube-api-access-mppvt\") pod \"network-check-target-jgtj4\" (UID: \"cb68da46-11b6-48f4-b302-3ad63d93f82f\") " pod="openshift-network-diagnostics/network-check-target-jgtj4" Oct 30 17:51:51.190491 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:51.189810 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:51:51.190491 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:51.189828 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:51:51.190491 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:51.189840 2579 projected.go:196] Error preparing data for projected volume kube-api-access-mppvt for pod openshift-network-diagnostics/network-check-target-jgtj4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:51:51.190491 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:51.189895 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb68da46-11b6-48f4-b302-3ad63d93f82f-kube-api-access-mppvt podName:cb68da46-11b6-48f4-b302-3ad63d93f82f nodeName:}" failed. No retries permitted until 2026-10-30 17:51:53.189875245 +0000 UTC m=+6.208832419 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mppvt" (UniqueName: "kubernetes.io/projected/cb68da46-11b6-48f4-b302-3ad63d93f82f-kube-api-access-mppvt") pod "network-check-target-jgtj4" (UID: "cb68da46-11b6-48f4-b302-3ad63d93f82f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:51:51.190491 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:51.190294 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:51:51.190491 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:51.190340 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fb157ba-42c2-4efc-a76b-6863fc987fb6-original-pull-secret podName:1fb157ba-42c2-4efc-a76b-6863fc987fb6 nodeName:}" failed. No retries permitted until 2026-10-30 17:51:53.190325944 +0000 UTC m=+6.209283114 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1fb157ba-42c2-4efc-a76b-6863fc987fb6-original-pull-secret") pod "global-pull-secret-syncer-pxv85" (UID: "1fb157ba-42c2-4efc-a76b-6863fc987fb6") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:51:51.190491 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:51.190422 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:51:51.190491 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:51.190457 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13521cae-dc8a-4670-9321-61ad6ab3fd4d-metrics-certs podName:13521cae-dc8a-4670-9321-61ad6ab3fd4d nodeName:}" failed. No retries permitted until 2026-10-30 17:51:53.190445632 +0000 UTC m=+6.209402801 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13521cae-dc8a-4670-9321-61ad6ab3fd4d-metrics-certs") pod "network-metrics-daemon-stxf2" (UID: "13521cae-dc8a-4670-9321-61ad6ab3fd4d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:51:51.424272 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:51.424091 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:46:49 +0000 UTC" deadline="2028-04-29 17:44:34.759214878 +0000 UTC" Oct 30 17:51:51.424272 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:51.424155 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13127h52m43.335064463s" Oct 30 17:51:51.500536 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:51.499998 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgtj4" Oct 30 17:51:51.500536 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:51.500138 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgtj4" podUID="cb68da46-11b6-48f4-b302-3ad63d93f82f" Oct 30 17:51:51.631632 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:51.631512 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal" event={"ID":"5ed893d4ed91b2f423e686d98f2bd57e","Type":"ContainerStarted","Data":"c3a9ad863613fae5f119ec157dea356a423b29cb4571a1efc14bddc8dc0e02ff"} Oct 30 17:51:51.677197 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:51.677076 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal" podStartSLOduration=2.677052222 podStartE2EDuration="2.677052222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:51:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:51.676182309 +0000 UTC m=+4.695139490" watchObservedRunningTime="2026-10-30 17:51:51.677052222 +0000 UTC m=+4.696009403" Oct 30 17:51:51.677456 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:51.677424 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-84.ec2.internal" podStartSLOduration=2.677411 podStartE2EDuration="2.677411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:51:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:50.620324703 +0000 UTC m=+3.639281894" watchObservedRunningTime="2026-10-30 17:51:51.677411 +0000 UTC m=+4.696368169" Oct 30 17:51:52.498827 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:52.498791 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:51:52.499337 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:52.498940 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxv85" podUID="1fb157ba-42c2-4efc-a76b-6863fc987fb6" Oct 30 17:51:52.499337 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:52.498791 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stxf2" Oct 30 17:51:52.499500 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:52.499350 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stxf2" podUID="13521cae-dc8a-4670-9321-61ad6ab3fd4d" Oct 30 17:51:53.209906 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:53.208990 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1fb157ba-42c2-4efc-a76b-6863fc987fb6-original-pull-secret\") pod \"global-pull-secret-syncer-pxv85\" (UID: \"1fb157ba-42c2-4efc-a76b-6863fc987fb6\") " pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:51:53.209906 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:53.209041 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13521cae-dc8a-4670-9321-61ad6ab3fd4d-metrics-certs\") pod \"network-metrics-daemon-stxf2\" (UID: \"13521cae-dc8a-4670-9321-61ad6ab3fd4d\") " pod="openshift-multus/network-metrics-daemon-stxf2" Oct 30 17:51:53.209906 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:53.209062 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mppvt\" (UniqueName: \"kubernetes.io/projected/cb68da46-11b6-48f4-b302-3ad63d93f82f-kube-api-access-mppvt\") pod \"network-check-target-jgtj4\" (UID: \"cb68da46-11b6-48f4-b302-3ad63d93f82f\") " pod="openshift-network-diagnostics/network-check-target-jgtj4" Oct 30 17:51:53.209906 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:53.209197 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:51:53.209906 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:53.209215 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:51:53.209906 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:53.209227 2579 projected.go:196] Error preparing data for projected volume kube-api-access-mppvt for pod openshift-network-diagnostics/network-check-target-jgtj4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:51:53.209906 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:53.209280 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb68da46-11b6-48f4-b302-3ad63d93f82f-kube-api-access-mppvt podName:cb68da46-11b6-48f4-b302-3ad63d93f82f nodeName:}" failed. No retries permitted until 2026-10-30 17:51:57.209265079 +0000 UTC m=+10.228222238 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mppvt" (UniqueName: "kubernetes.io/projected/cb68da46-11b6-48f4-b302-3ad63d93f82f-kube-api-access-mppvt") pod "network-check-target-jgtj4" (UID: "cb68da46-11b6-48f4-b302-3ad63d93f82f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:51:53.209906 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:53.209695 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:51:53.209906 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:53.209748 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fb157ba-42c2-4efc-a76b-6863fc987fb6-original-pull-secret podName:1fb157ba-42c2-4efc-a76b-6863fc987fb6 nodeName:}" failed. No retries permitted until 2026-10-30 17:51:57.209731304 +0000 UTC m=+10.228688464 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1fb157ba-42c2-4efc-a76b-6863fc987fb6-original-pull-secret") pod "global-pull-secret-syncer-pxv85" (UID: "1fb157ba-42c2-4efc-a76b-6863fc987fb6") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:51:53.209906 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:53.209806 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:51:53.209906 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:53.209841 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13521cae-dc8a-4670-9321-61ad6ab3fd4d-metrics-certs podName:13521cae-dc8a-4670-9321-61ad6ab3fd4d nodeName:}" failed. No retries permitted until 2026-10-30 17:51:57.209828615 +0000 UTC m=+10.228785775 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13521cae-dc8a-4670-9321-61ad6ab3fd4d-metrics-certs") pod "network-metrics-daemon-stxf2" (UID: "13521cae-dc8a-4670-9321-61ad6ab3fd4d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:51:53.498527 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:53.498493 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgtj4" Oct 30 17:51:53.498714 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:53.498602 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgtj4" podUID="cb68da46-11b6-48f4-b302-3ad63d93f82f" Oct 30 17:51:53.656819 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:53.656779 2579 generic.go:356] "Generic (PLEG): container finished" podID="cbda8c42-0f24-41a3-8a5f-727a4094d021" containerID="676de4088f0bdb1023774358ac66b2ad608fe36ff6c3645ed67ed420fbf91333" exitCode=0 Oct 30 17:51:53.657295 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:53.656837 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4bkwl" event={"ID":"cbda8c42-0f24-41a3-8a5f-727a4094d021","Type":"ContainerDied","Data":"676de4088f0bdb1023774358ac66b2ad608fe36ff6c3645ed67ed420fbf91333"} Oct 30 17:51:54.498978 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:54.498937 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:51:54.499171 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:54.499089 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxv85" podUID="1fb157ba-42c2-4efc-a76b-6863fc987fb6" Oct 30 17:51:54.499536 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:54.499511 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stxf2" Oct 30 17:51:54.499643 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:54.499625 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stxf2" podUID="13521cae-dc8a-4670-9321-61ad6ab3fd4d" Oct 30 17:51:54.663322 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:54.663278 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qzhll" event={"ID":"73dbe36c-9ccb-47d6-b11e-afc35b87008e","Type":"ContainerStarted","Data":"586aa080fdc0220503b8deef5d5856849128a017af332acedcaa5ed3194829de"} Oct 30 17:51:54.708770 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:54.708712 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qzhll" podStartSLOduration=2.81671741 podStartE2EDuration="6.708697878s" totalImagesPullingTime="3.891980468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:51:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:54.708424849 +0000 UTC m=+7.727382029" watchObservedRunningTime="2026-10-30 17:51:54.708697878 +0000 UTC m=+7.727655056" Oct 30 17:51:55.500877 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:55.500377 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgtj4" Oct 30 17:51:55.500877 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:55.500500 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgtj4" podUID="cb68da46-11b6-48f4-b302-3ad63d93f82f" Oct 30 17:51:55.671820 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:55.671750 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k75vq" event={"ID":"75cac6ab-2f87-4813-80a1-cc6efb29d187","Type":"ContainerStarted","Data":"04d90e54620a04cbe53fac63223c9065f4402f62574da354a154537460c37a65"} Oct 30 17:51:55.713077 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:55.712689 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-k75vq" podStartSLOduration=2.673172938 podStartE2EDuration="7.712668256s" totalImagesPullingTime="5.039495318s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:51:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:55.711684675 +0000 UTC m=+8.730641854" watchObservedRunningTime="2026-10-30 17:51:55.712668256 +0000 UTC m=+8.731625432" Oct 30 17:51:55.885671 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:55.885644 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qzhll" Oct 30 17:51:55.893401 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:55.886596 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qzhll" Oct 30 17:51:56.499240 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:56.499208 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stxf2" Oct 30 17:51:56.499432 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:56.499373 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stxf2" podUID="13521cae-dc8a-4670-9321-61ad6ab3fd4d" Oct 30 17:51:56.499841 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:56.499818 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:51:56.499949 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:56.499933 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxv85" podUID="1fb157ba-42c2-4efc-a76b-6863fc987fb6" Oct 30 17:51:56.677790 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:56.677756 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" event={"ID":"3b7db1b0-8abd-47fb-9df8-7caab8adfed6","Type":"ContainerStarted","Data":"e6903024f7a532e8a49975f5750e06455fb2d557775c91c2ddd0fc91818d18b2"} Oct 30 17:51:57.249741 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:57.249712 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mppvt\" (UniqueName: \"kubernetes.io/projected/cb68da46-11b6-48f4-b302-3ad63d93f82f-kube-api-access-mppvt\") pod \"network-check-target-jgtj4\" (UID: \"cb68da46-11b6-48f4-b302-3ad63d93f82f\") " pod="openshift-network-diagnostics/network-check-target-jgtj4" Oct 30 17:51:57.249857 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:57.249773 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1fb157ba-42c2-4efc-a76b-6863fc987fb6-original-pull-secret\") pod \"global-pull-secret-syncer-pxv85\" (UID: \"1fb157ba-42c2-4efc-a76b-6863fc987fb6\") " pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:51:57.249857 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:57.249794 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13521cae-dc8a-4670-9321-61ad6ab3fd4d-metrics-certs\") pod \"network-metrics-daemon-stxf2\" (UID: \"13521cae-dc8a-4670-9321-61ad6ab3fd4d\") " pod="openshift-multus/network-metrics-daemon-stxf2" Oct 30 17:51:57.249919 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:57.249879 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:51:57.249948 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:57.249923 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13521cae-dc8a-4670-9321-61ad6ab3fd4d-metrics-certs podName:13521cae-dc8a-4670-9321-61ad6ab3fd4d nodeName:}" failed. No retries permitted until 2026-10-30 17:52:05.249910871 +0000 UTC m=+18.268868038 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13521cae-dc8a-4670-9321-61ad6ab3fd4d-metrics-certs") pod "network-metrics-daemon-stxf2" (UID: "13521cae-dc8a-4670-9321-61ad6ab3fd4d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:51:57.250275 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:57.250229 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:51:57.250275 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:57.250245 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:51:57.250275 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:57.250255 2579 projected.go:196] Error preparing data for projected volume kube-api-access-mppvt for pod openshift-network-diagnostics/network-check-target-jgtj4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:51:57.250429 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:57.250288 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb68da46-11b6-48f4-b302-3ad63d93f82f-kube-api-access-mppvt podName:cb68da46-11b6-48f4-b302-3ad63d93f82f nodeName:}" failed. No retries permitted until 2026-10-30 17:52:05.250277935 +0000 UTC m=+18.269235101 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mppvt" (UniqueName: "kubernetes.io/projected/cb68da46-11b6-48f4-b302-3ad63d93f82f-kube-api-access-mppvt") pod "network-check-target-jgtj4" (UID: "cb68da46-11b6-48f4-b302-3ad63d93f82f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:51:57.250429 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:57.250355 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:51:57.250429 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:57.250387 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fb157ba-42c2-4efc-a76b-6863fc987fb6-original-pull-secret podName:1fb157ba-42c2-4efc-a76b-6863fc987fb6 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:05.250379331 +0000 UTC m=+18.269336498 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1fb157ba-42c2-4efc-a76b-6863fc987fb6-original-pull-secret") pod "global-pull-secret-syncer-pxv85" (UID: "1fb157ba-42c2-4efc-a76b-6863fc987fb6") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:51:57.292978 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:57.292949 2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 17:51:57.428575 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:57.427776 2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T17:51:57.29296859Z","UUID":"3134657d-8398-4296-a085-1020aa1dbbe7","Handler":null,"Name":"","Endpoint":""} Oct 30 17:51:57.430315 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:57.429731 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 17:51:57.430315 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:57.429760 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 17:51:57.500487 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:57.499980 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgtj4" Oct 30 17:51:57.500487 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:57.500111 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgtj4" podUID="cb68da46-11b6-48f4-b302-3ad63d93f82f" Oct 30 17:51:57.684179 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:57.683832 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sgp8n" event={"ID":"63b2e2e0-805f-4197-81d5-6eafe3831ded","Type":"ContainerStarted","Data":"8fadc1ff938327d3230019f71468d226010857d25b5c823850d74acded3fead1"} Oct 30 17:51:57.687771 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:57.687218 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:51:57.687771 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:57.687688 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" event={"ID":"3b7db1b0-8abd-47fb-9df8-7caab8adfed6","Type":"ContainerStarted","Data":"54cd627cc5d0af3da4cb5e850275c6a07718e97a1b96e91a1379ac2b18a18f96"} Oct 30 17:51:57.727903 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:57.727835 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-sgp8n" podStartSLOduration=1.933176151 podStartE2EDuration="9.72781581s" totalImagesPullingTime="7.794639659s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:51:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:57.727483888 +0000 UTC m=+10.746441063" watchObservedRunningTime="2026-10-30 17:51:57.72781581 +0000 UTC m=+10.746772984" Oct 30 17:51:58.498572 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:58.498536 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stxf2" Oct 30 17:51:58.498714 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:58.498691 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stxf2" podUID="13521cae-dc8a-4670-9321-61ad6ab3fd4d" Oct 30 17:51:58.498770 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:58.498735 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:51:58.498882 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:58.498858 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxv85" podUID="1fb157ba-42c2-4efc-a76b-6863fc987fb6" Oct 30 17:51:58.691911 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:58.691824 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bs5gk" event={"ID":"e1397e4e-e615-4954-8486-af559b65687e","Type":"ContainerStarted","Data":"908a7dc9334f8485a680d0cc792ab3bed34842c61c71ff70d92dcda04ff72faa"} Oct 30 17:51:58.694701 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:58.693994 2579 generic.go:356] "Generic (PLEG): container finished" podID="cbda8c42-0f24-41a3-8a5f-727a4094d021" containerID="72a2e9c81a427f06c4f14bc4e3da6a7a517968ecc4100a3b9ad687ec60937fb9" exitCode=0 Oct 30 17:51:58.694701 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:58.694086 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4bkwl" event={"ID":"cbda8c42-0f24-41a3-8a5f-727a4094d021","Type":"ContainerDied","Data":"72a2e9c81a427f06c4f14bc4e3da6a7a517968ecc4100a3b9ad687ec60937fb9"} Oct 30 17:51:58.697405 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:58.697376 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" event={"ID":"3b7db1b0-8abd-47fb-9df8-7caab8adfed6","Type":"ContainerStarted","Data":"13ff26ea6d4d825360b45afac461fc742b21e1406e30b5ea7aefc09bca38f0f5"} Oct 30 17:51:58.731497 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:58.731444 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-bs5gk" podStartSLOduration=2.576927929 podStartE2EDuration="10.731429355s" totalImagesPullingTime="8.154501426s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:51:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:58.7310786 +0000 UTC m=+11.750035781" watchObservedRunningTime="2026-10-30 17:51:58.731429355 +0000 UTC m=+11.750386531" Oct 30 17:51:58.773824 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:58.773763 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" podStartSLOduration=2.513135222 podStartE2EDuration="10.773743661s" totalImagesPullingTime="8.260608439s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:51:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:58.773372413 +0000 UTC m=+11.792329589" watchObservedRunningTime="2026-10-30 17:51:58.773743661 +0000 UTC m=+11.792700837" Oct 30 17:51:59.499118 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:59.498871 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgtj4" Oct 30 17:51:59.499262 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:51:59.499212 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgtj4" podUID="cb68da46-11b6-48f4-b302-3ad63d93f82f" Oct 30 17:51:59.701695 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:59.701656 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6826k" event={"ID":"8f348173-6e5e-404a-9126-7aa648f7eb01","Type":"ContainerStarted","Data":"293c565aa7d4a5cd71494d6167ef7fcb1abdaa602c62da10c9c494816ebe96d1"} Oct 30 17:51:59.741462 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:51:59.741402 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6826k" podStartSLOduration=3.646858517 podStartE2EDuration="11.741387921s" totalImagesPullingTime="8.094529404s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:51:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:59.740962422 +0000 UTC m=+12.759919598" watchObservedRunningTime="2026-10-30 17:51:59.741387921 +0000 UTC m=+12.760345096" Oct 30 17:52:00.498594 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:00.498554 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stxf2" Oct 30 17:52:00.498794 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:00.498605 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:52:00.498794 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:00.498703 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stxf2" podUID="13521cae-dc8a-4670-9321-61ad6ab3fd4d" Oct 30 17:52:00.498892 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:00.498835 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxv85" podUID="1fb157ba-42c2-4efc-a76b-6863fc987fb6" Oct 30 17:52:00.705684 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:00.705643 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t7knk" event={"ID":"c126d55e-cb37-4070-904d-47f76d781751","Type":"ContainerStarted","Data":"0ac46390c05f699c0043dade8ee170d39d11aa1ac487366adc27163295da774e"} Oct 30 17:52:00.708443 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:00.708409 2579 generic.go:356] "Generic (PLEG): container finished" podID="cbda8c42-0f24-41a3-8a5f-727a4094d021" containerID="d5396f27e8740f28710a9f671a35555e6aee1cb83b54c82fd8574d0fb3cd5554" exitCode=0 Oct 30 17:52:00.708596 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:00.708524 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4bkwl" event={"ID":"cbda8c42-0f24-41a3-8a5f-727a4094d021","Type":"ContainerDied","Data":"d5396f27e8740f28710a9f671a35555e6aee1cb83b54c82fd8574d0fb3cd5554"} Oct 30 17:52:00.748528 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:00.748474 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-t7knk" podStartSLOduration=2.486788284 podStartE2EDuration="12.748459993s" totalImagesPullingTime="10.261671709s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:51:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:00.748018348 +0000 UTC m=+13.766975525" watchObservedRunningTime="2026-10-30 17:52:00.748459993 +0000 UTC m=+13.767417170" Oct 30 17:52:01.499874 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:01.498194 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgtj4" Oct 30 17:52:01.499874 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:01.498340 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgtj4" podUID="cb68da46-11b6-48f4-b302-3ad63d93f82f" Oct 30 17:52:01.713766 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:01.713731 2579 generic.go:356] "Generic (PLEG): container finished" podID="cbda8c42-0f24-41a3-8a5f-727a4094d021" containerID="7116774dce2cb7873e3327c79f608296cca6a2a6ef34a9c1d19d57753bb2091f" exitCode=0 Oct 30 17:52:01.714428 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:01.713809 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4bkwl" event={"ID":"cbda8c42-0f24-41a3-8a5f-727a4094d021","Type":"ContainerDied","Data":"7116774dce2cb7873e3327c79f608296cca6a2a6ef34a9c1d19d57753bb2091f"} Oct 30 17:52:02.498434 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:02.498397 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stxf2" Oct 30 17:52:02.498609 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:02.498401 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:52:02.498609 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:02.498562 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stxf2" podUID="13521cae-dc8a-4670-9321-61ad6ab3fd4d" Oct 30 17:52:02.498609 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:02.498599 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxv85" podUID="1fb157ba-42c2-4efc-a76b-6863fc987fb6" Oct 30 17:52:03.498827 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:03.498787 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgtj4" Oct 30 17:52:03.499474 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:03.498934 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgtj4" podUID="cb68da46-11b6-48f4-b302-3ad63d93f82f" Oct 30 17:52:03.722184 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:03.721968 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4bkwl" event={"ID":"cbda8c42-0f24-41a3-8a5f-727a4094d021","Type":"ContainerStarted","Data":"c4189f47f1700b01c969b9ed31403e2f3ebf028a83c5dd56709c34dc67f095fc"} Oct 30 17:52:04.498777 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:04.498753 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stxf2" Oct 30 17:52:04.498918 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:04.498783 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:52:04.498918 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:04.498894 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stxf2" podUID="13521cae-dc8a-4670-9321-61ad6ab3fd4d" Oct 30 17:52:04.499224 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:04.499021 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxv85" podUID="1fb157ba-42c2-4efc-a76b-6863fc987fb6" Oct 30 17:52:04.726521 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:04.726489 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" event={"ID":"f983c48d-08c4-41e2-af03-7c36cf651faa","Type":"ContainerStarted","Data":"2c1edc1fa51d7a139bec31419a707e91bc7e397fd7c8afbe761a101bf4a06721"} Oct 30 17:52:04.726629 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:04.726563 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" event={"ID":"f983c48d-08c4-41e2-af03-7c36cf651faa","Type":"ContainerStarted","Data":"232ea6e9e33248711a5dc3971d10037d24f49a031a187dcedea10d0e7bb46c67"} Oct 30 17:52:04.726629 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:04.726599 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" event={"ID":"f983c48d-08c4-41e2-af03-7c36cf651faa","Type":"ContainerStarted","Data":"2db338b3f5485a04c7c961cb93d9dcf720585cedc848a5ebca8ace9b4fdaeef3"} Oct 30 17:52:04.726746 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:04.726642 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" event={"ID":"f983c48d-08c4-41e2-af03-7c36cf651faa","Type":"ContainerStarted","Data":"ffca287481c3e9bb2d28fe460a287ab3317c7d29f1a59fb07980643b23a292af"} Oct 30 17:52:04.726746 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:04.726682 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" event={"ID":"f983c48d-08c4-41e2-af03-7c36cf651faa","Type":"ContainerStarted","Data":"89eeefaa5582988828c62606d61d3c12d7d7da5f0563906edc976851258d5702"} Oct 30 17:52:04.729703 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:04.729674 2579 generic.go:356] "Generic (PLEG): container finished" podID="cbda8c42-0f24-41a3-8a5f-727a4094d021" containerID="c4189f47f1700b01c969b9ed31403e2f3ebf028a83c5dd56709c34dc67f095fc" exitCode=0 Oct 30 17:52:04.729785 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:04.729730 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4bkwl" event={"ID":"cbda8c42-0f24-41a3-8a5f-727a4094d021","Type":"ContainerDied","Data":"c4189f47f1700b01c969b9ed31403e2f3ebf028a83c5dd56709c34dc67f095fc"} Oct 30 17:52:05.310959 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:05.310741 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13521cae-dc8a-4670-9321-61ad6ab3fd4d-metrics-certs\") pod \"network-metrics-daemon-stxf2\" (UID: \"13521cae-dc8a-4670-9321-61ad6ab3fd4d\") " pod="openshift-multus/network-metrics-daemon-stxf2" Oct 30 17:52:05.311137 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:05.310980 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mppvt\" (UniqueName: \"kubernetes.io/projected/cb68da46-11b6-48f4-b302-3ad63d93f82f-kube-api-access-mppvt\") pod \"network-check-target-jgtj4\" (UID: \"cb68da46-11b6-48f4-b302-3ad63d93f82f\") " pod="openshift-network-diagnostics/network-check-target-jgtj4" Oct 30 17:52:05.311137 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:05.310884 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:52:05.311137 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:05.311037 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1fb157ba-42c2-4efc-a76b-6863fc987fb6-original-pull-secret\") pod \"global-pull-secret-syncer-pxv85\" (UID: \"1fb157ba-42c2-4efc-a76b-6863fc987fb6\") " pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:52:05.311137 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:05.311098 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13521cae-dc8a-4670-9321-61ad6ab3fd4d-metrics-certs podName:13521cae-dc8a-4670-9321-61ad6ab3fd4d nodeName:}" failed. No retries permitted until 2026-10-30 17:52:21.311076605 +0000 UTC m=+34.330033777 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13521cae-dc8a-4670-9321-61ad6ab3fd4d-metrics-certs") pod "network-metrics-daemon-stxf2" (UID: "13521cae-dc8a-4670-9321-61ad6ab3fd4d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:52:05.311137 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:05.311129 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:52:05.311339 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:05.311150 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:52:05.311339 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:05.311165 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:52:05.311339 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:05.311169 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fb157ba-42c2-4efc-a76b-6863fc987fb6-original-pull-secret podName:1fb157ba-42c2-4efc-a76b-6863fc987fb6 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:21.311158337 +0000 UTC m=+34.330115507 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1fb157ba-42c2-4efc-a76b-6863fc987fb6-original-pull-secret") pod "global-pull-secret-syncer-pxv85" (UID: "1fb157ba-42c2-4efc-a76b-6863fc987fb6") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:52:05.311339 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:05.311174 2579 projected.go:196] Error preparing data for projected volume kube-api-access-mppvt for pod openshift-network-diagnostics/network-check-target-jgtj4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:52:05.311339 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:05.311211 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb68da46-11b6-48f4-b302-3ad63d93f82f-kube-api-access-mppvt podName:cb68da46-11b6-48f4-b302-3ad63d93f82f nodeName:}" failed. No retries permitted until 2026-10-30 17:52:21.311201066 +0000 UTC m=+34.330158234 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mppvt" (UniqueName: "kubernetes.io/projected/cb68da46-11b6-48f4-b302-3ad63d93f82f-kube-api-access-mppvt") pod "network-check-target-jgtj4" (UID: "cb68da46-11b6-48f4-b302-3ad63d93f82f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:52:05.498309 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:05.498283 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgtj4" Oct 30 17:52:05.498491 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:05.498399 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgtj4" podUID="cb68da46-11b6-48f4-b302-3ad63d93f82f" Oct 30 17:52:05.734696 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:05.734665 2579 generic.go:356] "Generic (PLEG): container finished" podID="cbda8c42-0f24-41a3-8a5f-727a4094d021" containerID="7ad9ed6de9de0aa9a5bc4eb6b9c192579c9616fce9ededfacdfc012a2f1cd4a2" exitCode=0 Oct 30 17:52:05.735196 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:05.734759 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4bkwl" event={"ID":"cbda8c42-0f24-41a3-8a5f-727a4094d021","Type":"ContainerDied","Data":"7ad9ed6de9de0aa9a5bc4eb6b9c192579c9616fce9ededfacdfc012a2f1cd4a2"} Oct 30 17:52:05.737776 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:05.737710 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" event={"ID":"f983c48d-08c4-41e2-af03-7c36cf651faa","Type":"ContainerStarted","Data":"8e63911244932b6c15da9543dc159c52ac70db47b60c5f8b888798fc6b731a70"} Oct 30 17:52:06.498438 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:06.498401 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stxf2" Oct 30 17:52:06.498438 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:06.498427 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:52:06.498639 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:06.498515 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stxf2" podUID="13521cae-dc8a-4670-9321-61ad6ab3fd4d" Oct 30 17:52:06.498639 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:06.498588 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxv85" podUID="1fb157ba-42c2-4efc-a76b-6863fc987fb6" Oct 30 17:52:06.743303 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:06.743272 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4bkwl" event={"ID":"cbda8c42-0f24-41a3-8a5f-727a4094d021","Type":"ContainerStarted","Data":"98b5cc801bd8ec720224782013db5616616c9838424df1b04ed90abf0688f798"} Oct 30 17:52:06.815440 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:06.815302 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4bkwl" podStartSLOduration=8.120906198 podStartE2EDuration="18.815287722s" totalImagesPullingTime="10.694381524s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:51:48 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:06.814819468 +0000 UTC m=+19.833776645" watchObservedRunningTime="2026-10-30 17:52:06.815287722 +0000 UTC m=+19.834245088" Oct 30 17:52:07.501983 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:07.501955 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgtj4" Oct 30 17:52:07.502143 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:07.502049 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgtj4" podUID="cb68da46-11b6-48f4-b302-3ad63d93f82f" Oct 30 17:52:07.749207 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:07.749169 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" event={"ID":"f983c48d-08c4-41e2-af03-7c36cf651faa","Type":"ContainerStarted","Data":"c2f2f16bb54d725ed66dffeb6d5de3e26f130b628841189a4940374812e29640"} Oct 30 17:52:08.498547 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:08.498504 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stxf2" Oct 30 17:52:08.498725 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:08.498622 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stxf2" podUID="13521cae-dc8a-4670-9321-61ad6ab3fd4d" Oct 30 17:52:08.498725 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:08.498678 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:52:08.498792 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:08.498772 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxv85" podUID="1fb157ba-42c2-4efc-a76b-6863fc987fb6" Oct 30 17:52:09.501547 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:09.501517 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgtj4" Oct 30 17:52:09.502243 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:09.501625 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgtj4" podUID="cb68da46-11b6-48f4-b302-3ad63d93f82f" Oct 30 17:52:09.756360 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:09.756255 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" event={"ID":"f983c48d-08c4-41e2-af03-7c36cf651faa","Type":"ContainerStarted","Data":"a07e9d70d1a68a0913264646f5312b3a1a8393f01da682b60c9cf049663f6cd3"} Oct 30 17:52:09.756663 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:09.756546 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:09.773246 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:09.773214 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:09.846065 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:09.846017 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" podStartSLOduration=7.139145728 podStartE2EDuration="21.846003698s" totalImagesPullingTime="14.70685797s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:51:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:09.833161453 +0000 UTC m=+22.852118628" watchObservedRunningTime="2026-10-30 17:52:09.846003698 +0000 UTC m=+22.864960873" Oct 30 17:52:10.498838 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:10.498647 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stxf2" Oct 30 17:52:10.498838 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:10.498688 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:52:10.499012 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:10.498949 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxv85" podUID="1fb157ba-42c2-4efc-a76b-6863fc987fb6" Oct 30 17:52:10.499085 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:10.499069 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stxf2" podUID="13521cae-dc8a-4670-9321-61ad6ab3fd4d" Oct 30 17:52:10.759466 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:10.759372 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:10.759908 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:10.759481 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:10.775947 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:10.775920 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:11.165370 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:11.165225 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pxv85"] Oct 30 17:52:11.165566 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:11.165370 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:52:11.165566 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:11.165476 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxv85" podUID="1fb157ba-42c2-4efc-a76b-6863fc987fb6" Oct 30 17:52:11.168081 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:11.168050 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jgtj4"] Oct 30 17:52:11.168306 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:11.168181 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgtj4" Oct 30 17:52:11.168306 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:11.168288 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgtj4" podUID="cb68da46-11b6-48f4-b302-3ad63d93f82f" Oct 30 17:52:11.168861 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:11.168834 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-stxf2"] Oct 30 17:52:11.168972 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:11.168918 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stxf2" Oct 30 17:52:11.169037 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:11.169007 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stxf2" podUID="13521cae-dc8a-4670-9321-61ad6ab3fd4d" Oct 30 17:52:12.498731 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:12.498693 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stxf2" Oct 30 17:52:12.498731 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:12.498724 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:52:12.499270 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:12.498724 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgtj4" Oct 30 17:52:12.499270 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:12.498829 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stxf2" podUID="13521cae-dc8a-4670-9321-61ad6ab3fd4d" Oct 30 17:52:12.499270 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:12.498876 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxv85" podUID="1fb157ba-42c2-4efc-a76b-6863fc987fb6" Oct 30 17:52:12.499270 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:12.498934 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgtj4" podUID="cb68da46-11b6-48f4-b302-3ad63d93f82f" Oct 30 17:52:13.756140 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:13.756106 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qzhll" Oct 30 17:52:13.756618 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:13.756237 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:52:13.756936 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:13.756910 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qzhll" Oct 30 17:52:14.498309 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:14.498274 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stxf2" Oct 30 17:52:14.498527 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:14.498346 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgtj4" Oct 30 17:52:14.498527 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:14.498453 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:52:14.498527 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:14.498476 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stxf2" podUID="13521cae-dc8a-4670-9321-61ad6ab3fd4d" Oct 30 17:52:14.498527 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:14.498519 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxv85" podUID="1fb157ba-42c2-4efc-a76b-6863fc987fb6" Oct 30 17:52:14.498701 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:14.498562 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgtj4" podUID="cb68da46-11b6-48f4-b302-3ad63d93f82f" Oct 30 17:52:16.264101 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.264076 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-84.ec2.internal" event="NodeReady" Oct 30 17:52:16.264494 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.264192 2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 17:52:16.339055 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.339014 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jvc4q"] Oct 30 17:52:16.341908 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.341891 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:16.343215 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.343192 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jpjk4"] Oct 30 17:52:16.345902 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.345885 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jpjk4" Oct 30 17:52:16.372517 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.372488 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-p96b8"] Oct 30 17:52:16.375250 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.375224 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-p96b8" Oct 30 17:52:16.375378 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.375292 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 17:52:16.375378 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.375292 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 17:52:16.375378 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.375315 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cxbhw\"" Oct 30 17:52:16.384061 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.384044 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6xmc6\"" Oct 30 17:52:16.405130 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.405105 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jpjk4"] Oct 30 17:52:16.423224 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.423193 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-p96b8"] Oct 30 17:52:16.453003 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.452954 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 17:52:16.453003 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.452999 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 17:52:16.453174 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.453074 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 17:52:16.453174 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.453144 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 17:52:16.453259 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.453185 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 17:52:16.453259 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.453215 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 17:52:16.461065 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.461041 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jvc4q"] Oct 30 17:52:16.471477 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.471456 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 17:52:16.471591 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.471501 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-99rd9\"" Oct 30 17:52:16.484311 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.484285 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b38a6e02-c51f-455b-b5bd-25ca428cddb2-crio-socket\") pod \"insights-runtime-extractor-jvc4q\" (UID: \"b38a6e02-c51f-455b-b5bd-25ca428cddb2\") " pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:16.484311 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.484312 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mww6s\" (UniqueName: \"kubernetes.io/projected/60c3d534-02fc-4665-ada2-e56fba531b87-kube-api-access-mww6s\") pod \"ingress-canary-p96b8\" (UID: \"60c3d534-02fc-4665-ada2-e56fba531b87\") " pod="openshift-ingress-canary/ingress-canary-p96b8" Oct 30 17:52:16.484525 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.484354 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b38a6e02-c51f-455b-b5bd-25ca428cddb2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvc4q\" (UID: \"b38a6e02-c51f-455b-b5bd-25ca428cddb2\") " pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:16.484525 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.484438 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b38a6e02-c51f-455b-b5bd-25ca428cddb2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvc4q\" (UID: \"b38a6e02-c51f-455b-b5bd-25ca428cddb2\") " pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:16.484525 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.484469 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b38a6e02-c51f-455b-b5bd-25ca428cddb2-data-volume\") pod \"insights-runtime-extractor-jvc4q\" (UID: \"b38a6e02-c51f-455b-b5bd-25ca428cddb2\") " pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:16.484525 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.484489 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7lk6\" (UniqueName: \"kubernetes.io/projected/b38a6e02-c51f-455b-b5bd-25ca428cddb2-kube-api-access-n7lk6\") pod \"insights-runtime-extractor-jvc4q\" (UID: \"b38a6e02-c51f-455b-b5bd-25ca428cddb2\") " pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:16.484698 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.484583 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e23597-79e7-4a28-ab52-fc8b343226ab-config-volume\") pod \"dns-default-jpjk4\" (UID: \"c9e23597-79e7-4a28-ab52-fc8b343226ab\") " pod="openshift-dns/dns-default-jpjk4" Oct 30 17:52:16.484698 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.484662 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9e23597-79e7-4a28-ab52-fc8b343226ab-tmp-dir\") pod \"dns-default-jpjk4\" (UID: \"c9e23597-79e7-4a28-ab52-fc8b343226ab\") " pod="openshift-dns/dns-default-jpjk4" Oct 30 17:52:16.484698 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.484688 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9e23597-79e7-4a28-ab52-fc8b343226ab-metrics-tls\") pod \"dns-default-jpjk4\" (UID: \"c9e23597-79e7-4a28-ab52-fc8b343226ab\") " pod="openshift-dns/dns-default-jpjk4" Oct 30 17:52:16.484794 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.484729 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60c3d534-02fc-4665-ada2-e56fba531b87-cert\") pod \"ingress-canary-p96b8\" (UID: \"60c3d534-02fc-4665-ada2-e56fba531b87\") " pod="openshift-ingress-canary/ingress-canary-p96b8" Oct 30 17:52:16.484794 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.484746 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnxpn\" (UniqueName: \"kubernetes.io/projected/c9e23597-79e7-4a28-ab52-fc8b343226ab-kube-api-access-lnxpn\") pod \"dns-default-jpjk4\" (UID: \"c9e23597-79e7-4a28-ab52-fc8b343226ab\") " pod="openshift-dns/dns-default-jpjk4" Oct 30 17:52:16.498457 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.498433 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgtj4" Oct 30 17:52:16.498580 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.498544 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stxf2" Oct 30 17:52:16.498750 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.498723 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:52:16.553086 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.553057 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 17:52:16.553086 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.553057 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 17:52:16.572712 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.572683 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 17:52:16.572712 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.572707 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zlfnf\"" Oct 30 17:52:16.572872 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.572786 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-x8qzn\"" Oct 30 17:52:16.572872 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.572800 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ddhr5\"" Oct 30 17:52:16.572872 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.572809 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 17:52:16.585497 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.585476 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b38a6e02-c51f-455b-b5bd-25ca428cddb2-crio-socket\") pod \"insights-runtime-extractor-jvc4q\" (UID: \"b38a6e02-c51f-455b-b5bd-25ca428cddb2\") " pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:16.585566 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.585499 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mww6s\" (UniqueName: \"kubernetes.io/projected/60c3d534-02fc-4665-ada2-e56fba531b87-kube-api-access-mww6s\") pod \"ingress-canary-p96b8\" (UID: \"60c3d534-02fc-4665-ada2-e56fba531b87\") " pod="openshift-ingress-canary/ingress-canary-p96b8" Oct 30 17:52:16.585566 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.585535 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b38a6e02-c51f-455b-b5bd-25ca428cddb2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvc4q\" (UID: \"b38a6e02-c51f-455b-b5bd-25ca428cddb2\") " pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:16.585566 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.585559 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b38a6e02-c51f-455b-b5bd-25ca428cddb2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvc4q\" (UID: \"b38a6e02-c51f-455b-b5bd-25ca428cddb2\") " pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:16.585700 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.585589 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b38a6e02-c51f-455b-b5bd-25ca428cddb2-data-volume\") pod \"insights-runtime-extractor-jvc4q\" (UID: \"b38a6e02-c51f-455b-b5bd-25ca428cddb2\") " pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:16.585700 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.585612 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7lk6\" (UniqueName: \"kubernetes.io/projected/b38a6e02-c51f-455b-b5bd-25ca428cddb2-kube-api-access-n7lk6\") pod \"insights-runtime-extractor-jvc4q\" (UID: \"b38a6e02-c51f-455b-b5bd-25ca428cddb2\") " pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:16.585700 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.585639 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e23597-79e7-4a28-ab52-fc8b343226ab-config-volume\") pod \"dns-default-jpjk4\" (UID: \"c9e23597-79e7-4a28-ab52-fc8b343226ab\") " pod="openshift-dns/dns-default-jpjk4" Oct 30 17:52:16.585700 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:16.585661 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 17:52:16.585700 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.585674 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9e23597-79e7-4a28-ab52-fc8b343226ab-tmp-dir\") pod \"dns-default-jpjk4\" (UID: \"c9e23597-79e7-4a28-ab52-fc8b343226ab\") " pod="openshift-dns/dns-default-jpjk4" Oct 30 17:52:16.585700 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:16.585680 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:52:16.585700 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.585697 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9e23597-79e7-4a28-ab52-fc8b343226ab-metrics-tls\") pod \"dns-default-jpjk4\" (UID: \"c9e23597-79e7-4a28-ab52-fc8b343226ab\") " pod="openshift-dns/dns-default-jpjk4" Oct 30 17:52:16.585979 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:16.585727 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b38a6e02-c51f-455b-b5bd-25ca428cddb2-kube-rbac-proxy-cm podName:b38a6e02-c51f-455b-b5bd-25ca428cddb2 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:17.085709151 +0000 UTC m=+30.104666327 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b38a6e02-c51f-455b-b5bd-25ca428cddb2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jvc4q" (UID: "b38a6e02-c51f-455b-b5bd-25ca428cddb2") : configmap "kube-rbac-proxy" not found Oct 30 17:52:16.585979 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.585760 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60c3d534-02fc-4665-ada2-e56fba531b87-cert\") pod \"ingress-canary-p96b8\" (UID: \"60c3d534-02fc-4665-ada2-e56fba531b87\") " pod="openshift-ingress-canary/ingress-canary-p96b8" Oct 30 17:52:16.585979 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:16.585777 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:52:16.585979 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.585785 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnxpn\" (UniqueName: \"kubernetes.io/projected/c9e23597-79e7-4a28-ab52-fc8b343226ab-kube-api-access-lnxpn\") pod \"dns-default-jpjk4\" (UID: \"c9e23597-79e7-4a28-ab52-fc8b343226ab\") " pod="openshift-dns/dns-default-jpjk4" Oct 30 17:52:16.585979 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.585723 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b38a6e02-c51f-455b-b5bd-25ca428cddb2-crio-socket\") pod \"insights-runtime-extractor-jvc4q\" (UID: \"b38a6e02-c51f-455b-b5bd-25ca428cddb2\") " pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:16.585979 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:16.585823 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9e23597-79e7-4a28-ab52-fc8b343226ab-metrics-tls podName:c9e23597-79e7-4a28-ab52-fc8b343226ab nodeName:}" failed. No retries permitted until 2026-10-30 17:52:17.085812341 +0000 UTC m=+30.104769514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c9e23597-79e7-4a28-ab52-fc8b343226ab-metrics-tls") pod "dns-default-jpjk4" (UID: "c9e23597-79e7-4a28-ab52-fc8b343226ab") : secret "dns-default-metrics-tls" not found Oct 30 17:52:16.585979 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:16.585934 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b38a6e02-c51f-455b-b5bd-25ca428cddb2-insights-runtime-extractor-tls podName:b38a6e02-c51f-455b-b5bd-25ca428cddb2 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:17.085913434 +0000 UTC m=+30.104870602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b38a6e02-c51f-455b-b5bd-25ca428cddb2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jvc4q" (UID: "b38a6e02-c51f-455b-b5bd-25ca428cddb2") : secret "insights-runtime-extractor-tls" not found Oct 30 17:52:16.586238 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.586041 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b38a6e02-c51f-455b-b5bd-25ca428cddb2-data-volume\") pod \"insights-runtime-extractor-jvc4q\" (UID: \"b38a6e02-c51f-455b-b5bd-25ca428cddb2\") " pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:16.586238 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.586216 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9e23597-79e7-4a28-ab52-fc8b343226ab-tmp-dir\") pod \"dns-default-jpjk4\" (UID: \"c9e23597-79e7-4a28-ab52-fc8b343226ab\") " pod="openshift-dns/dns-default-jpjk4" Oct 30 17:52:16.586427 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.586405 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e23597-79e7-4a28-ab52-fc8b343226ab-config-volume\") pod \"dns-default-jpjk4\" (UID: \"c9e23597-79e7-4a28-ab52-fc8b343226ab\") " pod="openshift-dns/dns-default-jpjk4" Oct 30 17:52:16.589608 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.589587 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60c3d534-02fc-4665-ada2-e56fba531b87-cert\") pod \"ingress-canary-p96b8\" (UID: \"60c3d534-02fc-4665-ada2-e56fba531b87\") " pod="openshift-ingress-canary/ingress-canary-p96b8" Oct 30 17:52:16.609450 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.609421 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mww6s\" (UniqueName: \"kubernetes.io/projected/60c3d534-02fc-4665-ada2-e56fba531b87-kube-api-access-mww6s\") pod \"ingress-canary-p96b8\" (UID: \"60c3d534-02fc-4665-ada2-e56fba531b87\") " pod="openshift-ingress-canary/ingress-canary-p96b8" Oct 30 17:52:16.610369 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.610349 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7lk6\" (UniqueName: \"kubernetes.io/projected/b38a6e02-c51f-455b-b5bd-25ca428cddb2-kube-api-access-n7lk6\") pod \"insights-runtime-extractor-jvc4q\" (UID: \"b38a6e02-c51f-455b-b5bd-25ca428cddb2\") " pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:16.610510 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.610495 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnxpn\" (UniqueName: \"kubernetes.io/projected/c9e23597-79e7-4a28-ab52-fc8b343226ab-kube-api-access-lnxpn\") pod \"dns-default-jpjk4\" (UID: \"c9e23597-79e7-4a28-ab52-fc8b343226ab\") " pod="openshift-dns/dns-default-jpjk4" Oct 30 17:52:16.685310 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.685274 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-p96b8" Oct 30 17:52:16.993425 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:16.993236 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-p96b8"] Oct 30 17:52:16.995431 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:52:16.995405 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c3d534_02fc_4665_ada2_e56fba531b87.slice/crio-b78aa875ef7cb538c5307aa35e52712ad1b773d813d0833e51e5d96437354016 WatchSource:0}: Error finding container b78aa875ef7cb538c5307aa35e52712ad1b773d813d0833e51e5d96437354016: Status 404 returned error can't find the container with id b78aa875ef7cb538c5307aa35e52712ad1b773d813d0833e51e5d96437354016 Oct 30 17:52:17.089344 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:17.089285 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9e23597-79e7-4a28-ab52-fc8b343226ab-metrics-tls\") pod \"dns-default-jpjk4\" (UID: \"c9e23597-79e7-4a28-ab52-fc8b343226ab\") " pod="openshift-dns/dns-default-jpjk4" Oct 30 17:52:17.089485 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:17.089388 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b38a6e02-c51f-455b-b5bd-25ca428cddb2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvc4q\" (UID: \"b38a6e02-c51f-455b-b5bd-25ca428cddb2\") " pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:17.089485 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:17.089418 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b38a6e02-c51f-455b-b5bd-25ca428cddb2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvc4q\" (UID: \"b38a6e02-c51f-455b-b5bd-25ca428cddb2\") " pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:17.089485 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:17.089435 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:52:17.089685 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:17.089494 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9e23597-79e7-4a28-ab52-fc8b343226ab-metrics-tls podName:c9e23597-79e7-4a28-ab52-fc8b343226ab nodeName:}" failed. No retries permitted until 2026-10-30 17:52:18.089479847 +0000 UTC m=+31.108437028 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c9e23597-79e7-4a28-ab52-fc8b343226ab-metrics-tls") pod "dns-default-jpjk4" (UID: "c9e23597-79e7-4a28-ab52-fc8b343226ab") : secret "dns-default-metrics-tls" not found Oct 30 17:52:17.089685 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:17.089501 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:52:17.089685 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:17.089503 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 17:52:17.089685 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:17.089544 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b38a6e02-c51f-455b-b5bd-25ca428cddb2-insights-runtime-extractor-tls podName:b38a6e02-c51f-455b-b5bd-25ca428cddb2 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:18.08953314 +0000 UTC m=+31.108490315 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b38a6e02-c51f-455b-b5bd-25ca428cddb2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jvc4q" (UID: "b38a6e02-c51f-455b-b5bd-25ca428cddb2") : secret "insights-runtime-extractor-tls" not found Oct 30 17:52:17.089685 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:17.089557 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b38a6e02-c51f-455b-b5bd-25ca428cddb2-kube-rbac-proxy-cm podName:b38a6e02-c51f-455b-b5bd-25ca428cddb2 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:18.089550756 +0000 UTC m=+31.108507909 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b38a6e02-c51f-455b-b5bd-25ca428cddb2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jvc4q" (UID: "b38a6e02-c51f-455b-b5bd-25ca428cddb2") : configmap "kube-rbac-proxy" not found Oct 30 17:52:17.776577 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:17.776536 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-p96b8" event={"ID":"60c3d534-02fc-4665-ada2-e56fba531b87","Type":"ContainerStarted","Data":"b78aa875ef7cb538c5307aa35e52712ad1b773d813d0833e51e5d96437354016"} Oct 30 17:52:18.098864 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:18.098774 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b38a6e02-c51f-455b-b5bd-25ca428cddb2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvc4q\" (UID: \"b38a6e02-c51f-455b-b5bd-25ca428cddb2\") " pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:18.098864 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:18.098846 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9e23597-79e7-4a28-ab52-fc8b343226ab-metrics-tls\") pod \"dns-default-jpjk4\" (UID: \"c9e23597-79e7-4a28-ab52-fc8b343226ab\") " pod="openshift-dns/dns-default-jpjk4" Oct 30 17:52:18.099098 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:18.098919 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b38a6e02-c51f-455b-b5bd-25ca428cddb2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvc4q\" (UID: \"b38a6e02-c51f-455b-b5bd-25ca428cddb2\") " pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:18.099098 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:18.098945 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:52:18.099098 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:18.099001 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 17:52:18.099098 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:18.099029 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b38a6e02-c51f-455b-b5bd-25ca428cddb2-insights-runtime-extractor-tls podName:b38a6e02-c51f-455b-b5bd-25ca428cddb2 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:20.099008924 +0000 UTC m=+33.117966083 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b38a6e02-c51f-455b-b5bd-25ca428cddb2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jvc4q" (UID: "b38a6e02-c51f-455b-b5bd-25ca428cddb2") : secret "insights-runtime-extractor-tls" not found Oct 30 17:52:18.099098 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:18.099030 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:52:18.099098 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:18.099061 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b38a6e02-c51f-455b-b5bd-25ca428cddb2-kube-rbac-proxy-cm podName:b38a6e02-c51f-455b-b5bd-25ca428cddb2 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:20.099037528 +0000 UTC m=+33.117994698 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b38a6e02-c51f-455b-b5bd-25ca428cddb2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jvc4q" (UID: "b38a6e02-c51f-455b-b5bd-25ca428cddb2") : configmap "kube-rbac-proxy" not found Oct 30 17:52:18.099098 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:18.099091 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9e23597-79e7-4a28-ab52-fc8b343226ab-metrics-tls podName:c9e23597-79e7-4a28-ab52-fc8b343226ab nodeName:}" failed. No retries permitted until 2026-10-30 17:52:20.099075419 +0000 UTC m=+33.118032586 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c9e23597-79e7-4a28-ab52-fc8b343226ab-metrics-tls") pod "dns-default-jpjk4" (UID: "c9e23597-79e7-4a28-ab52-fc8b343226ab") : secret "dns-default-metrics-tls" not found Oct 30 17:52:19.570241 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:19.570211 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bs5gk_e1397e4e-e615-4954-8486-af559b65687e/dns-node-resolver/0.log" Oct 30 17:52:19.658846 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:19.658807 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qzhll_73dbe36c-9ccb-47d6-b11e-afc35b87008e/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:19.782393 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:19.782359 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-p96b8" event={"ID":"60c3d534-02fc-4665-ada2-e56fba531b87","Type":"ContainerStarted","Data":"e950593d17c9b5c0e71c52acaf4be59143e31a8e56677ff99d4054296b0c6e6c"} Oct 30 17:52:19.826438 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:19.826339 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-p96b8" podStartSLOduration=1.142530782 podStartE2EDuration="3.826309288s" totalImagesPullingTime="2.683778506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:52:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:19.825273372 +0000 UTC m=+32.844230549" watchObservedRunningTime="2026-10-30 17:52:19.826309288 +0000 UTC m=+32.845266462" Oct 30 17:52:19.829218 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:19.829193 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qzhll_73dbe36c-9ccb-47d6-b11e-afc35b87008e/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:20.117095 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:20.117008 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b38a6e02-c51f-455b-b5bd-25ca428cddb2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvc4q\" (UID: \"b38a6e02-c51f-455b-b5bd-25ca428cddb2\") " pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:20.117095 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:20.117060 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b38a6e02-c51f-455b-b5bd-25ca428cddb2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvc4q\" (UID: \"b38a6e02-c51f-455b-b5bd-25ca428cddb2\") " pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:20.117438 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:20.117108 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9e23597-79e7-4a28-ab52-fc8b343226ab-metrics-tls\") pod \"dns-default-jpjk4\" (UID: \"c9e23597-79e7-4a28-ab52-fc8b343226ab\") " pod="openshift-dns/dns-default-jpjk4" Oct 30 17:52:20.117438 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:20.117127 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 17:52:20.117438 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:20.117188 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:52:20.117438 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:20.117206 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b38a6e02-c51f-455b-b5bd-25ca428cddb2-kube-rbac-proxy-cm podName:b38a6e02-c51f-455b-b5bd-25ca428cddb2 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:24.117188563 +0000 UTC m=+37.136145733 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b38a6e02-c51f-455b-b5bd-25ca428cddb2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jvc4q" (UID: "b38a6e02-c51f-455b-b5bd-25ca428cddb2") : configmap "kube-rbac-proxy" not found Oct 30 17:52:20.117438 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:20.117227 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:52:20.117438 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:20.117239 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b38a6e02-c51f-455b-b5bd-25ca428cddb2-insights-runtime-extractor-tls podName:b38a6e02-c51f-455b-b5bd-25ca428cddb2 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:24.11722641 +0000 UTC m=+37.136183562 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b38a6e02-c51f-455b-b5bd-25ca428cddb2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jvc4q" (UID: "b38a6e02-c51f-455b-b5bd-25ca428cddb2") : secret "insights-runtime-extractor-tls" not found Oct 30 17:52:20.117438 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:20.117298 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9e23597-79e7-4a28-ab52-fc8b343226ab-metrics-tls podName:c9e23597-79e7-4a28-ab52-fc8b343226ab nodeName:}" failed. No retries permitted until 2026-10-30 17:52:24.11728513 +0000 UTC m=+37.136242297 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c9e23597-79e7-4a28-ab52-fc8b343226ab-metrics-tls") pod "dns-default-jpjk4" (UID: "c9e23597-79e7-4a28-ab52-fc8b343226ab") : secret "dns-default-metrics-tls" not found Oct 30 17:52:20.200046 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:20.200007 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qzhll_73dbe36c-9ccb-47d6-b11e-afc35b87008e/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:20.390281 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:20.390194 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qzhll_73dbe36c-9ccb-47d6-b11e-afc35b87008e/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:20.969601 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:20.969572 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k75vq_75cac6ab-2f87-4813-80a1-cc6efb29d187/node-ca/0.log" Oct 30 17:52:21.328410 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:21.328376 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1fb157ba-42c2-4efc-a76b-6863fc987fb6-original-pull-secret\") pod \"global-pull-secret-syncer-pxv85\" (UID: \"1fb157ba-42c2-4efc-a76b-6863fc987fb6\") " pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:52:21.328617 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:21.328427 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13521cae-dc8a-4670-9321-61ad6ab3fd4d-metrics-certs\") pod \"network-metrics-daemon-stxf2\" (UID: \"13521cae-dc8a-4670-9321-61ad6ab3fd4d\") " pod="openshift-multus/network-metrics-daemon-stxf2" Oct 30 17:52:21.328617 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:21.328451 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mppvt\" (UniqueName: \"kubernetes.io/projected/cb68da46-11b6-48f4-b302-3ad63d93f82f-kube-api-access-mppvt\") pod \"network-check-target-jgtj4\" (UID: \"cb68da46-11b6-48f4-b302-3ad63d93f82f\") " pod="openshift-network-diagnostics/network-check-target-jgtj4" Oct 30 17:52:21.328617 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:21.328557 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 17:52:21.328779 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:21.328637 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13521cae-dc8a-4670-9321-61ad6ab3fd4d-metrics-certs podName:13521cae-dc8a-4670-9321-61ad6ab3fd4d nodeName:}" failed. No retries permitted until 2026-10-30 17:52:53.328617713 +0000 UTC m=+66.347574872 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13521cae-dc8a-4670-9321-61ad6ab3fd4d-metrics-certs") pod "network-metrics-daemon-stxf2" (UID: "13521cae-dc8a-4670-9321-61ad6ab3fd4d") : secret "metrics-daemon-secret" not found Oct 30 17:52:21.331252 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:21.331235 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1fb157ba-42c2-4efc-a76b-6863fc987fb6-original-pull-secret\") pod \"global-pull-secret-syncer-pxv85\" (UID: \"1fb157ba-42c2-4efc-a76b-6863fc987fb6\") " pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:52:21.331616 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:21.331594 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mppvt\" (UniqueName: \"kubernetes.io/projected/cb68da46-11b6-48f4-b302-3ad63d93f82f-kube-api-access-mppvt\") pod \"network-check-target-jgtj4\" (UID: \"cb68da46-11b6-48f4-b302-3ad63d93f82f\") " pod="openshift-network-diagnostics/network-check-target-jgtj4" Oct 30 17:52:21.570677 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:21.570649 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-p96b8_60c3d534-02fc-4665-ada2-e56fba531b87/serve-healthcheck-canary/0.log" Oct 30 17:52:21.610743 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:21.610650 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgtj4" Oct 30 17:52:21.619894 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:21.619860 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:52:22.007824 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:22.007796 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jgtj4"] Oct 30 17:52:22.009459 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:52:22.009425 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb68da46_11b6_48f4_b302_3ad63d93f82f.slice/crio-55e996343a802cdda6b8be5b128355933f273d87b7b858dbcc81d9d8c53a27e0 WatchSource:0}: Error finding container 55e996343a802cdda6b8be5b128355933f273d87b7b858dbcc81d9d8c53a27e0: Status 404 returned error can't find the container with id 55e996343a802cdda6b8be5b128355933f273d87b7b858dbcc81d9d8c53a27e0 Oct 30 17:52:22.027108 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:22.027086 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pxv85"] Oct 30 17:52:22.028982 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:52:22.028956 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fb157ba_42c2_4efc_a76b_6863fc987fb6.slice/crio-bcd3d19f28b9e870d6cafcb6ce6918d8f2537f72f27eacb23269e70d9c4b50cf WatchSource:0}: Error finding container bcd3d19f28b9e870d6cafcb6ce6918d8f2537f72f27eacb23269e70d9c4b50cf: Status 404 returned error can't find the container with id bcd3d19f28b9e870d6cafcb6ce6918d8f2537f72f27eacb23269e70d9c4b50cf Oct 30 17:52:22.791564 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:22.791524 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pxv85" event={"ID":"1fb157ba-42c2-4efc-a76b-6863fc987fb6","Type":"ContainerStarted","Data":"bcd3d19f28b9e870d6cafcb6ce6918d8f2537f72f27eacb23269e70d9c4b50cf"} Oct 30 17:52:22.792995 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:22.792962 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jgtj4" event={"ID":"cb68da46-11b6-48f4-b302-3ad63d93f82f","Type":"ContainerStarted","Data":"55e996343a802cdda6b8be5b128355933f273d87b7b858dbcc81d9d8c53a27e0"} Oct 30 17:52:24.149851 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:24.149813 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b38a6e02-c51f-455b-b5bd-25ca428cddb2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvc4q\" (UID: \"b38a6e02-c51f-455b-b5bd-25ca428cddb2\") " pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:24.150365 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:24.149871 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b38a6e02-c51f-455b-b5bd-25ca428cddb2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvc4q\" (UID: \"b38a6e02-c51f-455b-b5bd-25ca428cddb2\") " pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:24.150365 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:24.149908 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9e23597-79e7-4a28-ab52-fc8b343226ab-metrics-tls\") pod \"dns-default-jpjk4\" (UID: \"c9e23597-79e7-4a28-ab52-fc8b343226ab\") " pod="openshift-dns/dns-default-jpjk4" Oct 30 17:52:24.151079 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:24.150604 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:52:24.151079 ip-10-0-130-84 kubenswrapper[2579]: E1030 17:52:24.150701 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b38a6e02-c51f-455b-b5bd-25ca428cddb2-insights-runtime-extractor-tls podName:b38a6e02-c51f-455b-b5bd-25ca428cddb2 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:32.150678766 +0000 UTC m=+45.169635938 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b38a6e02-c51f-455b-b5bd-25ca428cddb2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jvc4q" (UID: "b38a6e02-c51f-455b-b5bd-25ca428cddb2") : secret "insights-runtime-extractor-tls" not found Oct 30 17:52:24.151079 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:24.150982 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b38a6e02-c51f-455b-b5bd-25ca428cddb2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvc4q\" (UID: \"b38a6e02-c51f-455b-b5bd-25ca428cddb2\") " pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:24.153082 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:24.153057 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9e23597-79e7-4a28-ab52-fc8b343226ab-metrics-tls\") pod \"dns-default-jpjk4\" (UID: \"c9e23597-79e7-4a28-ab52-fc8b343226ab\") " pod="openshift-dns/dns-default-jpjk4" Oct 30 17:52:24.162279 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:24.162248 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jpjk4" Oct 30 17:52:24.493808 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:24.493780 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jpjk4"] Oct 30 17:52:24.500086 ip-10-0-130-84 kubenswrapper[2579]: W1030 17:52:24.500049 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e23597_79e7_4a28_ab52_fc8b343226ab.slice/crio-0d5746ebfc1c8717d8e753c111dc4d7a251c1a24123463dcb3a9b263b11cfc49 WatchSource:0}: Error finding container 0d5746ebfc1c8717d8e753c111dc4d7a251c1a24123463dcb3a9b263b11cfc49: Status 404 returned error can't find the container with id 0d5746ebfc1c8717d8e753c111dc4d7a251c1a24123463dcb3a9b263b11cfc49 Oct 30 17:52:24.801900 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:24.801817 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jpjk4" event={"ID":"c9e23597-79e7-4a28-ab52-fc8b343226ab","Type":"ContainerStarted","Data":"0d5746ebfc1c8717d8e753c111dc4d7a251c1a24123463dcb3a9b263b11cfc49"} Oct 30 17:52:25.808271 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:25.808184 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jgtj4" event={"ID":"cb68da46-11b6-48f4-b302-3ad63d93f82f","Type":"ContainerStarted","Data":"c72cfa980a51c3a5e4b22c83e6bb0ffb6027f0366a65a76b40a9f1a3013dfc4a"} Oct 30 17:52:25.808756 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:25.808359 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jgtj4" Oct 30 17:52:25.855845 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:25.855779 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jgtj4" podStartSLOduration=34.359388236 podStartE2EDuration="37.85575864s" totalImagesPullingTime="3.496370404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:51:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:25.853716227 +0000 UTC m=+38.872673402" watchObservedRunningTime="2026-10-30 17:52:25.85575864 +0000 UTC m=+38.874715817" Oct 30 17:52:26.811779 ip-10-0-130-84 kubenswrapper[2579]: I1030 17:52:26.811749 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jpjk4" event={"ID":"c9e23597-79e7-4a28-ab52-fc8b343226ab","Type":"ContainerStarted","Data":"f0726b1a23c61f9938f8092c49af95c1989570d5768ccaa0a551f67b1f9f7b7a"} Oct 30 17:52:27.445667 ip-10-0-130-84 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 17:52:27.459829 ip-10-0-130-84 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 17:52:27.459997 ip-10-0-130-84 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 17:52:27.460044 ip-10-0-130-84 systemd[1]: kubelet.service: Consumed 2.375s CPU time. Oct 30 17:52:27.482022 ip-10-0-130-84 systemd[1]: Starting Kubernetes Kubelet... Oct 30 17:52:27.554106 ip-10-0-130-84 kubenswrapper[4455]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:52:27.554106 ip-10-0-130-84 kubenswrapper[4455]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 17:52:27.554106 ip-10-0-130-84 kubenswrapper[4455]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:52:27.554106 ip-10-0-130-84 kubenswrapper[4455]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:52:27.556366 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556268 4455 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:52:27.556366 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556366 4455 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:52:27.556442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556371 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:52:27.556442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556374 4455 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:52:27.556442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556377 4455 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:52:27.556442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556380 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:52:27.556442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556383 4455 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:52:27.556442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556385 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:52:27.556442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556388 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:52:27.556442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556391 4455 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:52:27.556442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556393 4455 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:52:27.556442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556396 4455 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:52:27.556442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556398 4455 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:52:27.556442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556401 4455 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:52:27.556442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556403 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:52:27.556442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556406 4455 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:52:27.556442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556409 4455 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:52:27.556442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556411 4455 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:52:27.556442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556414 4455 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:52:27.556442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556418 4455 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:52:27.556442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556434 4455 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:52:27.556936 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556439 4455 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:52:27.556936 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556443 4455 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:52:27.556936 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556446 4455 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:52:27.556936 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556449 4455 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:52:27.556936 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556452 4455 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:52:27.556936 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556455 4455 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:52:27.556936 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556458 4455 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:52:27.556936 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556469 4455 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:52:27.556936 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556472 4455 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:52:27.556936 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556475 4455 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:52:27.556936 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556477 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:52:27.556936 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556480 4455 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:52:27.556936 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556482 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:52:27.556936 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556485 4455 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:52:27.556936 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556488 4455 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:52:27.556936 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556490 4455 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:52:27.556936 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556493 4455 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:52:27.556936 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556495 4455 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:52:27.556936 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556497 4455 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:52:27.556936 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556500 4455 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:52:27.557453 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556502 4455 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:52:27.557453 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556505 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:52:27.557453 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556507 4455 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:52:27.557453 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556510 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:52:27.557453 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556512 4455 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:52:27.557453 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556515 4455 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:52:27.557453 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556517 4455 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:52:27.557453 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556520 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:52:27.557453 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556522 4455 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:52:27.557453 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556524 4455 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:52:27.557453 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556527 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:52:27.557453 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556530 4455 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:52:27.557453 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556533 4455 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:52:27.557453 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556536 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:52:27.557453 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556539 4455 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:52:27.557453 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556542 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:52:27.557453 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556544 4455 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:52:27.557453 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556547 4455 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:52:27.557453 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556550 4455 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:52:27.557915 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556552 4455 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:52:27.557915 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556555 4455 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:52:27.557915 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556558 4455 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:52:27.557915 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556562 4455 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:52:27.557915 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556567 4455 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:52:27.557915 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556570 4455 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:52:27.557915 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556572 4455 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:52:27.557915 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556575 4455 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:52:27.557915 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556577 4455 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:52:27.557915 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556580 4455 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:52:27.557915 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556582 4455 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:52:27.557915 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556585 4455 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:52:27.557915 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556587 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:52:27.557915 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556590 4455 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:52:27.557915 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556592 4455 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:52:27.557915 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556595 4455 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:52:27.557915 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556598 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:52:27.557915 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556600 4455 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:52:27.557915 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556604 4455 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:52:27.558440 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556608 4455 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:52:27.558440 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556611 4455 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:52:27.558440 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556614 4455 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:52:27.558440 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556617 4455 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:52:27.558440 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556624 4455 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:52:27.558440 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556626 4455 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:52:27.558440 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556630 4455 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:52:27.558440 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556636 4455 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:52:27.558440 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556640 4455 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:52:27.558440 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556644 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:52:27.558440 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556648 4455 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:52:27.558440 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556652 4455 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:52:27.558440 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556656 4455 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:52:27.558440 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556658 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:52:27.558440 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556661 4455 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:52:27.558440 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556664 4455 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:52:27.558440 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556667 4455 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:52:27.558440 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556670 4455 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:52:27.558440 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556673 4455 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:52:27.558909 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556675 4455 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:52:27.558909 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556679 4455 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:52:27.558909 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556681 4455 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:52:27.558909 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556684 4455 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:52:27.558909 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556686 4455 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:52:27.558909 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556689 4455 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:52:27.558909 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556691 4455 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:52:27.558909 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.556693 4455 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:52:27.558909 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557465 4455 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:52:27.558909 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557475 4455 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:52:27.558909 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557478 4455 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:52:27.558909 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557481 4455 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:52:27.558909 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557485 4455 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:52:27.558909 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557487 4455 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:52:27.558909 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557490 4455 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:52:27.558909 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557493 4455 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:52:27.558909 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557496 4455 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:52:27.558909 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557499 4455 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:52:27.558909 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557502 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:52:27.559390 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557504 4455 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:52:27.559390 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557507 4455 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:52:27.559390 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557509 4455 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:52:27.559390 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557512 4455 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:52:27.559390 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557514 4455 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:52:27.559390 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557517 4455 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:52:27.559390 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557519 4455 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:52:27.559390 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557522 4455 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:52:27.559390 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557524 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:52:27.559390 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557527 4455 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:52:27.559390 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557530 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:52:27.559390 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557532 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:52:27.559390 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557535 4455 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:52:27.559390 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557537 4455 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:52:27.559390 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557540 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:52:27.559390 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557542 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:52:27.559390 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557544 4455 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:52:27.559390 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557547 4455 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:52:27.559390 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557549 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:52:27.559390 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557551 4455 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:52:27.559869 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557554 4455 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:52:27.559869 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557556 4455 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:52:27.559869 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557559 4455 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:52:27.559869 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557561 4455 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:52:27.559869 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557564 4455 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:52:27.559869 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557566 4455 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:52:27.559869 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557569 4455 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:52:27.559869 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557572 4455 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:52:27.559869 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557574 4455 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:52:27.559869 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557577 4455 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:52:27.559869 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557580 4455 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:52:27.559869 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557582 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:52:27.559869 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557585 4455 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:52:27.559869 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557588 4455 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:52:27.559869 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557591 4455 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:52:27.559869 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557593 4455 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:52:27.559869 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557596 4455 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:52:27.559869 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557598 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:52:27.559869 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557601 4455 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:52:27.559869 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557603 4455 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:52:27.560590 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557606 4455 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:52:27.560590 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557608 4455 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:52:27.560590 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557611 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:52:27.560590 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557613 4455 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:52:27.560590 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557616 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:52:27.560590 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557619 4455 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:52:27.560590 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557621 4455 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:52:27.560590 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557623 4455 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:52:27.560590 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557626 4455 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:52:27.560590 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557628 4455 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:52:27.560590 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557631 4455 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:52:27.560590 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557634 4455 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:52:27.560590 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557636 4455 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:52:27.560590 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557638 4455 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:52:27.560590 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557647 4455 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:52:27.560590 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557650 4455 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:52:27.560590 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557652 4455 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:52:27.560590 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557654 4455 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:52:27.560590 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557657 4455 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:52:27.560590 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557659 4455 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:52:27.561143 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557662 4455 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:52:27.561143 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557665 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:52:27.561143 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557667 4455 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:52:27.561143 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557670 4455 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:52:27.561143 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557672 4455 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:52:27.561143 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557675 4455 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:52:27.561143 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557677 4455 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:52:27.561143 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557681 4455 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:52:27.561143 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557685 4455 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:52:27.561143 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557688 4455 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:52:27.561143 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557690 4455 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:52:27.561143 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557693 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:52:27.561143 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557695 4455 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:52:27.561143 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557698 4455 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:52:27.561143 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557701 4455 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:52:27.561143 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557703 4455 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:52:27.561143 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557707 4455 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:52:27.561143 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557709 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:52:27.561143 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557712 4455 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:52:27.561658 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557714 4455 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:52:27.561658 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557717 4455 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:52:27.561658 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557719 4455 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:52:27.561658 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557721 4455 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:52:27.561658 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557724 4455 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:52:27.561658 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557726 4455 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:52:27.561658 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557729 4455 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:52:27.561658 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557733 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:52:27.561658 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557735 4455 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:52:27.561658 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557738 4455 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:52:27.561658 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557740 4455 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:52:27.561658 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557743 4455 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:52:27.561658 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557745 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:52:27.561658 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557748 4455 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:52:27.561658 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557750 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:52:27.561658 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.557753 4455 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:52:27.561658 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558246 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:52:27.561658 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558251 4455 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:52:27.561658 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558254 4455 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:52:27.562111 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558257 4455 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:52:27.562111 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558259 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:52:27.562111 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558262 4455 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:52:27.562111 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558265 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:52:27.562111 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558268 4455 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:52:27.562111 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558270 4455 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:52:27.562111 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558273 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:52:27.562111 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558275 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:52:27.562111 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558278 4455 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:52:27.562111 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558280 4455 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:52:27.562111 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558283 4455 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:52:27.562111 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558286 4455 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:52:27.562111 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558289 4455 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:52:27.562111 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558291 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:52:27.562111 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558294 4455 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:52:27.562111 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558296 4455 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:52:27.562111 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558299 4455 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:52:27.562111 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558302 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:52:27.562111 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558304 4455 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:52:27.562578 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558308 4455 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:52:27.562578 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558311 4455 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:52:27.562578 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558314 4455 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:52:27.562578 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558335 4455 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:52:27.562578 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558339 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:52:27.562578 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558343 4455 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:52:27.562578 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558345 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:52:27.562578 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558348 4455 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:52:27.562578 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558351 4455 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:52:27.562578 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558353 4455 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:52:27.562578 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558356 4455 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:52:27.562578 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558358 4455 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:52:27.562578 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558361 4455 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:52:27.562578 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558364 4455 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:52:27.562578 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558366 4455 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:52:27.562578 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558369 4455 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:52:27.562578 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558371 4455 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:52:27.562578 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558374 4455 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:52:27.562578 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558376 4455 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:52:27.563046 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558379 4455 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:52:27.563046 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558381 4455 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:52:27.563046 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558384 4455 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:52:27.563046 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558386 4455 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:52:27.563046 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558389 4455 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:52:27.563046 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558392 4455 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:52:27.563046 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558395 4455 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:52:27.563046 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558397 4455 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:52:27.563046 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558400 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:52:27.563046 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558403 4455 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:52:27.563046 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558405 4455 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:52:27.563046 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558408 4455 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:52:27.563046 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558410 4455 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:52:27.563046 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558414 4455 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:52:27.563046 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558417 4455 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:52:27.563046 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558419 4455 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:52:27.563046 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558421 4455 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:52:27.563046 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558424 4455 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:52:27.563046 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558427 4455 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:52:27.563046 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558429 4455 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:52:27.563890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558431 4455 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:52:27.563890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558434 4455 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:52:27.563890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558436 4455 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:52:27.563890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558439 4455 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:52:27.563890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558442 4455 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:52:27.563890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558446 4455 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:52:27.563890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558449 4455 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:52:27.563890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558452 4455 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:52:27.563890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558455 4455 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:52:27.563890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558459 4455 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:52:27.563890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558462 4455 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:52:27.563890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558465 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:52:27.563890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558467 4455 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:52:27.563890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558469 4455 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:52:27.563890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558472 4455 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:52:27.563890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558474 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:52:27.563890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558477 4455 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:52:27.563890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558479 4455 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:52:27.563890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558483 4455 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:52:27.563890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558485 4455 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:52:27.564760 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558488 4455 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:52:27.564760 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558490 4455 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:52:27.564760 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558493 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:52:27.564760 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558495 4455 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:52:27.564760 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558498 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:52:27.564760 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558502 4455 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:52:27.564760 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558505 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:52:27.564760 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558508 4455 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:52:27.564760 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558510 4455 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:52:27.564760 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558513 4455 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:52:27.564760 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558515 4455 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:52:27.564760 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558518 4455 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:52:27.564760 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558520 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:52:27.564760 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558523 4455 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:52:27.564760 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558525 4455 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:52:27.564760 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558528 4455 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:52:27.564760 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558530 4455 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:52:27.564760 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558533 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:52:27.564760 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558536 4455 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:52:27.565581 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558539 4455 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:52:27.565581 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558541 4455 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:52:27.565581 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558544 4455 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:52:27.565581 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558546 4455 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:52:27.565581 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558549 4455 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:52:27.565581 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.558551 4455 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:52:27.565581 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558649 4455 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 17:52:27.565581 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558660 4455 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 17:52:27.565581 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558670 4455 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 17:52:27.565581 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558677 4455 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 17:52:27.565581 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558681 4455 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 17:52:27.565581 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558684 4455 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 17:52:27.565581 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558689 4455 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 17:52:27.565581 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558694 4455 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 17:52:27.565581 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558697 4455 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 17:52:27.565581 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558700 4455 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 17:52:27.565581 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558703 4455 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 17:52:27.565581 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558706 4455 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 17:52:27.565581 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558711 4455 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 17:52:27.565581 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558714 4455 flags.go:64] FLAG: --cgroup-root="" Oct 30 17:52:27.565581 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558716 4455 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 17:52:27.565581 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558719 4455 flags.go:64] FLAG: --client-ca-file="" Oct 30 17:52:27.565581 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558722 4455 flags.go:64] FLAG: --cloud-provider="external" Oct 30 17:52:27.566590 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558725 4455 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 17:52:27.566590 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558728 4455 flags.go:64] FLAG: --cluster-domain="" Oct 30 17:52:27.566590 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558731 4455 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 17:52:27.566590 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558734 4455 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 17:52:27.566590 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558737 4455 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 17:52:27.566590 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558740 4455 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 17:52:27.566590 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558746 4455 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 17:52:27.566590 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558749 4455 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 17:52:27.566590 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558752 4455 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 17:52:27.566590 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558755 4455 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 17:52:27.566590 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558758 4455 flags.go:64] FLAG: --contention-profiling="false" Oct 30 17:52:27.566590 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558760 4455 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 17:52:27.566590 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558763 4455 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 17:52:27.566590 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558766 4455 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 17:52:27.566590 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558769 4455 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 17:52:27.566590 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558773 4455 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 17:52:27.566590 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558776 4455 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 17:52:27.566590 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558779 4455 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 17:52:27.566590 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558782 4455 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 17:52:27.566590 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558785 4455 flags.go:64] FLAG: --enable-server="true" Oct 30 17:52:27.566590 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558787 4455 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 17:52:27.566590 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558791 4455 flags.go:64] FLAG: --event-burst="100" Oct 30 17:52:27.566590 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558795 4455 flags.go:64] FLAG: --event-qps="50" Oct 30 17:52:27.566590 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558798 4455 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 17:52:27.567698 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558801 4455 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 17:52:27.567698 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558804 4455 flags.go:64] FLAG: --eviction-hard="" Oct 30 17:52:27.567698 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558808 4455 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 17:52:27.567698 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558812 4455 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 17:52:27.567698 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558815 4455 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 17:52:27.567698 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558818 4455 flags.go:64] FLAG: --eviction-soft="" Oct 30 17:52:27.567698 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558821 4455 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 17:52:27.567698 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558824 4455 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 17:52:27.567698 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558826 4455 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 17:52:27.567698 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558829 4455 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 17:52:27.567698 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558832 4455 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 17:52:27.567698 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558836 4455 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 17:52:27.567698 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558838 4455 flags.go:64] FLAG: --feature-gates="" Oct 30 17:52:27.567698 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558845 4455 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 17:52:27.567698 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558850 4455 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 17:52:27.567698 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558853 4455 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 17:52:27.567698 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558856 4455 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 17:52:27.567698 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558859 4455 flags.go:64] FLAG: --healthz-port="10248" Oct 30 17:52:27.567698 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558862 4455 flags.go:64] FLAG: --help="false" Oct 30 17:52:27.567698 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558865 4455 flags.go:64] FLAG: --hostname-override="ip-10-0-130-84.ec2.internal" Oct 30 17:52:27.567698 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558868 4455 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 17:52:27.567698 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558871 4455 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 17:52:27.567698 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558874 4455 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 17:52:27.568738 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558879 4455 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 17:52:27.568738 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558882 4455 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 17:52:27.568738 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558885 4455 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 17:52:27.568738 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558888 4455 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 17:52:27.568738 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558891 4455 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 17:52:27.568738 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558894 4455 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 17:52:27.568738 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558896 4455 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 17:52:27.568738 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558900 4455 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 17:52:27.568738 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558903 4455 flags.go:64] FLAG: --kube-reserved="" Oct 30 17:52:27.568738 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558906 4455 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 17:52:27.568738 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558909 4455 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 17:52:27.568738 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558912 4455 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 17:52:27.568738 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558916 4455 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 17:52:27.568738 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558919 4455 flags.go:64] FLAG: --lock-file="" Oct 30 17:52:27.568738 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558922 4455 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 17:52:27.568738 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558925 4455 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 17:52:27.568738 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558928 4455 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 17:52:27.568738 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558934 4455 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 17:52:27.568738 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558936 4455 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 17:52:27.568738 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558939 4455 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 17:52:27.568738 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558942 4455 flags.go:64] FLAG: --logging-format="text" Oct 30 17:52:27.568738 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558944 4455 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 17:52:27.568738 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558948 4455 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 17:52:27.568738 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558951 4455 flags.go:64] FLAG: --manifest-url="" Oct 30 17:52:27.569763 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558954 4455 flags.go:64] FLAG: --manifest-url-header="" Oct 30 17:52:27.569763 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558959 4455 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 17:52:27.569763 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558962 4455 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 17:52:27.569763 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558966 4455 flags.go:64] FLAG: --max-pods="110" Oct 30 17:52:27.569763 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558969 4455 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 17:52:27.569763 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558972 4455 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 17:52:27.569763 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558974 4455 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 17:52:27.569763 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558977 4455 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 17:52:27.569763 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558980 4455 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 17:52:27.569763 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558983 4455 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 17:52:27.569763 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558986 4455 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 17:52:27.569763 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558992 4455 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 17:52:27.569763 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558995 4455 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 17:52:27.569763 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.558998 4455 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 17:52:27.569763 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559001 4455 flags.go:64] FLAG: --pod-cidr="" Oct 30 17:52:27.569763 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559004 4455 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 17:52:27.569763 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559007 4455 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 17:52:27.569763 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559010 4455 flags.go:64] FLAG: --pods-per-core="0" Oct 30 17:52:27.569763 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559013 4455 flags.go:64] FLAG: --port="10250" Oct 30 17:52:27.569763 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559016 4455 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 17:52:27.569763 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559020 4455 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c358448deb9909bd" Oct 30 17:52:27.569763 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559023 4455 flags.go:64] FLAG: --qos-reserved="" Oct 30 17:52:27.569763 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559026 4455 flags.go:64] FLAG: --read-only-port="10255" Oct 30 17:52:27.569763 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559029 4455 flags.go:64] FLAG: --register-node="true" Oct 30 17:52:27.569763 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559032 4455 flags.go:64] FLAG: --register-with-taints="" Oct 30 17:52:27.569763 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559043 4455 flags.go:64] FLAG: --registry-burst="10" Oct 30 17:52:27.571019 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559047 4455 flags.go:64] FLAG: --registry-qps="5" Oct 30 17:52:27.571019 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559049 4455 flags.go:64] FLAG: --reserved-cpus="" Oct 30 17:52:27.571019 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559052 4455 flags.go:64] FLAG: --reserved-memory="" Oct 30 17:52:27.571019 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559056 4455 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 17:52:27.571019 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559059 4455 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 17:52:27.571019 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559062 4455 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 17:52:27.571019 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559065 4455 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 17:52:27.571019 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559067 4455 flags.go:64] FLAG: --runonce="false" Oct 30 17:52:27.571019 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559070 4455 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 17:52:27.571019 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559073 4455 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 17:52:27.571019 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559076 4455 flags.go:64] FLAG: --seccomp-default="false" Oct 30 17:52:27.571019 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559079 4455 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 17:52:27.571019 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559081 4455 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 17:52:27.571019 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559084 4455 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 17:52:27.571019 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559087 4455 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 17:52:27.571019 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559090 4455 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 17:52:27.571019 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559093 4455 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 17:52:27.571019 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559095 4455 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 17:52:27.571019 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559098 4455 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 17:52:27.571019 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559101 4455 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 17:52:27.571019 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559104 4455 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 17:52:27.571019 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559106 4455 flags.go:64] FLAG: --system-cgroups="" Oct 30 17:52:27.571019 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559109 4455 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 17:52:27.571019 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559114 4455 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 17:52:27.571019 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559118 4455 flags.go:64] FLAG: --tls-cert-file="" Oct 30 17:52:27.572123 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559121 4455 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 17:52:27.572123 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559125 4455 flags.go:64] FLAG: --tls-min-version="" Oct 30 17:52:27.572123 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559128 4455 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 17:52:27.572123 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559130 4455 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 17:52:27.572123 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559133 4455 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 17:52:27.572123 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559136 4455 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 17:52:27.572123 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559139 4455 flags.go:64] FLAG: --v="2" Oct 30 17:52:27.572123 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559143 4455 flags.go:64] FLAG: --version="false" Oct 30 17:52:27.572123 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559147 4455 flags.go:64] FLAG: --vmodule="" Oct 30 17:52:27.572123 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559151 4455 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 17:52:27.572123 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559154 4455 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 17:52:27.572123 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559289 4455 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:52:27.572123 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559292 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:52:27.572123 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559295 4455 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:52:27.572123 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559297 4455 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:52:27.572123 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559300 4455 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:52:27.572123 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559303 4455 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:52:27.572123 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559305 4455 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:52:27.572123 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559308 4455 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:52:27.572123 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559310 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:52:27.572123 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559312 4455 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:52:27.572123 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559315 4455 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:52:27.572955 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559330 4455 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:52:27.572955 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559333 4455 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:52:27.572955 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559335 4455 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:52:27.572955 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559338 4455 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:52:27.572955 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559341 4455 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:52:27.572955 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559343 4455 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:52:27.572955 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559346 4455 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:52:27.572955 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559349 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:52:27.572955 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559352 4455 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:52:27.572955 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559354 4455 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:52:27.572955 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559357 4455 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:52:27.572955 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559361 4455 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:52:27.572955 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559363 4455 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:52:27.572955 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559366 4455 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:52:27.572955 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559369 4455 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:52:27.572955 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559371 4455 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:52:27.572955 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559374 4455 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:52:27.572955 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559377 4455 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:52:27.572955 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559379 4455 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:52:27.572955 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559381 4455 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:52:27.573496 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559384 4455 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:52:27.573496 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559386 4455 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:52:27.573496 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559389 4455 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:52:27.573496 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559392 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:52:27.573496 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559395 4455 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:52:27.573496 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559397 4455 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:52:27.573496 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559400 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:52:27.573496 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559404 4455 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:52:27.573496 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559408 4455 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:52:27.573496 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559411 4455 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:52:27.573496 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559413 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:52:27.573496 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559416 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:52:27.573496 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559418 4455 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:52:27.573496 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559421 4455 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:52:27.573496 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559423 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:52:27.573496 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559426 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:52:27.573496 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559428 4455 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:52:27.573496 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559431 4455 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:52:27.573496 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559433 4455 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:52:27.573948 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559435 4455 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:52:27.573948 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559438 4455 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:52:27.573948 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559440 4455 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:52:27.573948 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559444 4455 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:52:27.573948 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559448 4455 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:52:27.573948 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559450 4455 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:52:27.573948 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559453 4455 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:52:27.573948 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559456 4455 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:52:27.573948 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559458 4455 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:52:27.573948 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559461 4455 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:52:27.573948 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559463 4455 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:52:27.573948 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559465 4455 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:52:27.573948 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559468 4455 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:52:27.573948 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559470 4455 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:52:27.573948 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559473 4455 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:52:27.573948 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559475 4455 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:52:27.573948 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559478 4455 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:52:27.573948 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559480 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:52:27.573948 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559483 4455 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:52:27.574606 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559485 4455 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:52:27.574606 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559487 4455 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:52:27.574606 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559490 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:52:27.574606 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559492 4455 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:52:27.574606 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559495 4455 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:52:27.574606 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559497 4455 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:52:27.574606 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559499 4455 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:52:27.574606 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559502 4455 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:52:27.574606 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559504 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:52:27.574606 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559507 4455 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:52:27.574606 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559510 4455 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:52:27.574606 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559512 4455 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:52:27.574606 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559514 4455 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:52:27.574606 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559517 4455 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:52:27.574606 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559519 4455 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:52:27.574606 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559522 4455 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:52:27.574606 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559526 4455 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:52:27.574606 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559529 4455 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:52:27.574606 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559532 4455 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:52:27.574606 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559535 4455 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:52:27.575104 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559537 4455 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:52:27.575104 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559540 4455 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:52:27.575104 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559542 4455 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:52:27.575104 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559545 4455 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:52:27.575104 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559547 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:52:27.575104 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559550 4455 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:52:27.575104 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559552 4455 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:52:27.575104 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559555 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:52:27.575104 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559557 4455 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:52:27.575104 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559559 4455 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:52:27.575104 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559562 4455 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:52:27.575104 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559564 4455 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:52:27.575104 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559567 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:52:27.575104 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559569 4455 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:52:27.575104 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559573 4455 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:52:27.575104 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559577 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:52:27.575104 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559580 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:52:27.575620 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.559607 4455 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:52:27.575620 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559642 4455 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:52:27.575620 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559647 4455 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:52:27.575620 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559650 4455 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:52:27.575620 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559654 4455 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:52:27.575620 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559656 4455 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:52:27.575620 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559659 4455 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:52:27.575620 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559661 4455 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:52:27.575620 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559664 4455 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:52:27.575620 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559666 4455 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:52:27.575620 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559669 4455 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:52:27.575620 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559673 4455 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:52:27.575620 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559676 4455 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:52:27.575620 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559680 4455 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:52:27.575620 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559683 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:52:27.575620 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559686 4455 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:52:27.576084 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559688 4455 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:52:27.576084 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559691 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:52:27.576084 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559695 4455 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:52:27.576084 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559699 4455 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:52:27.576084 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559701 4455 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:52:27.576084 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559704 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:52:27.576084 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559707 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:52:27.576084 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559709 4455 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:52:27.576084 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559712 4455 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:52:27.576084 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559714 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:52:27.576084 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559717 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:52:27.576084 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559719 4455 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:52:27.576084 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559722 4455 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:52:27.576084 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559724 4455 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:52:27.576084 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559727 4455 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:52:27.576084 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559729 4455 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:52:27.576084 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559732 4455 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:52:27.576084 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559735 4455 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:52:27.576084 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559737 4455 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:52:27.576585 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559739 4455 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:52:27.576585 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559742 4455 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:52:27.576585 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559744 4455 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:52:27.576585 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559746 4455 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:52:27.576585 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559749 4455 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:52:27.576585 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559751 4455 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:52:27.576585 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559754 4455 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:52:27.576585 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559756 4455 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:52:27.576585 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559760 4455 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:52:27.576585 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559763 4455 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:52:27.576585 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559766 4455 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:52:27.576585 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559769 4455 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:52:27.576585 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559772 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:52:27.576585 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559774 4455 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:52:27.576585 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559777 4455 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:52:27.576585 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559780 4455 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:52:27.576585 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559783 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:52:27.576585 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559785 4455 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:52:27.576585 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559787 4455 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:52:27.576585 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559790 4455 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:52:27.577070 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559792 4455 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:52:27.577070 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559795 4455 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:52:27.577070 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559797 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:52:27.577070 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559801 4455 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:52:27.577070 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559803 4455 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:52:27.577070 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559806 4455 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:52:27.577070 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559808 4455 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:52:27.577070 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559811 4455 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:52:27.577070 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559813 4455 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:52:27.577070 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559816 4455 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:52:27.577070 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559818 4455 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:52:27.577070 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559821 4455 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:52:27.577070 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559823 4455 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:52:27.577070 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559825 4455 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:52:27.577070 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559828 4455 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:52:27.577070 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559830 4455 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:52:27.577070 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559833 4455 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:52:27.577070 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559835 4455 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:52:27.577070 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559838 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:52:27.577070 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559840 4455 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:52:27.577609 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559844 4455 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:52:27.577609 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559846 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:52:27.577609 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559850 4455 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:52:27.577609 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559860 4455 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:52:27.577609 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559863 4455 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:52:27.577609 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.559892 4455 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:52:27.577609 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.560427 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:52:27.577609 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.560447 4455 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:52:27.577609 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.560457 4455 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:52:27.577609 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.560467 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:52:27.577609 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.560472 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:52:27.577609 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.560478 4455 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:52:27.577609 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.560483 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:52:27.577609 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.560487 4455 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:52:27.577609 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.560492 4455 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:52:27.577609 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.560497 4455 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:52:27.577609 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.560503 4455 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:52:27.577609 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.560507 4455 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:52:27.577609 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.560520 4455 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:52:27.578364 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.560525 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:52:27.578364 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.560529 4455 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:52:27.578364 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.560534 4455 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:52:27.578364 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.560538 4455 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:52:27.578364 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.560543 4455 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:52:27.578364 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.560547 4455 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:52:27.578364 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.560551 4455 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:52:27.578364 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.560556 4455 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:52:27.578364 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.560560 4455 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:52:27.578364 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.560569 4455 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:52:27.578364 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.560573 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:52:27.578364 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.560583 4455 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:52:27.578364 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.560589 4455 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:52:27.578364 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.565660 4455 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 17:52:27.578364 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.565678 4455 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 17:52:27.578364 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.565703 4455 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 17:52:27.578364 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.565710 4455 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 17:52:27.578364 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565805 4455 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:52:27.578364 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565814 4455 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:52:27.579022 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565820 4455 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:52:27.579022 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565825 4455 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:52:27.579022 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565829 4455 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:52:27.579022 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565834 4455 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:52:27.579022 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565838 4455 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:52:27.579022 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565842 4455 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:52:27.579022 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565847 4455 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:52:27.579022 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565851 4455 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:52:27.579022 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565855 4455 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:52:27.579022 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565860 4455 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:52:27.579022 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565864 4455 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:52:27.579022 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565869 4455 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:52:27.579022 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565873 4455 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:52:27.579022 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565877 4455 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:52:27.579022 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565881 4455 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:52:27.579022 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565886 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:52:27.579022 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565892 4455 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:52:27.579022 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565899 4455 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:52:27.579022 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565904 4455 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:52:27.579626 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565909 4455 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:52:27.579626 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565913 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:52:27.579626 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565917 4455 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:52:27.579626 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565922 4455 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:52:27.579626 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565926 4455 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:52:27.579626 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565931 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:52:27.579626 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565935 4455 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:52:27.579626 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565946 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:52:27.579626 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565951 4455 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:52:27.579626 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565956 4455 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:52:27.579626 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565960 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:52:27.579626 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565964 4455 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:52:27.579626 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565968 4455 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:52:27.579626 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565972 4455 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:52:27.579626 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565977 4455 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:52:27.579626 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565981 4455 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:52:27.579626 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565985 4455 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:52:27.579626 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565990 4455 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:52:27.579626 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565994 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:52:27.580074 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.565998 4455 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:52:27.580074 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566002 4455 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:52:27.580074 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566006 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:52:27.580074 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566010 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:52:27.580074 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566015 4455 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:52:27.580074 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566019 4455 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:52:27.580074 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566025 4455 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:52:27.580074 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566029 4455 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:52:27.580074 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566033 4455 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:52:27.580074 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566038 4455 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:52:27.580074 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566042 4455 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:52:27.580074 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566046 4455 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:52:27.580074 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566052 4455 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:52:27.580074 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566058 4455 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:52:27.580074 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566062 4455 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:52:27.580074 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566067 4455 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:52:27.580074 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566072 4455 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:52:27.580074 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566077 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:52:27.580074 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566081 4455 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:52:27.580074 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566086 4455 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:52:27.580584 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566090 4455 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:52:27.580584 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566094 4455 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:52:27.580584 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566098 4455 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:52:27.580584 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566102 4455 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:52:27.580584 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566106 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:52:27.580584 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566111 4455 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:52:27.580584 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566115 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:52:27.580584 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566119 4455 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:52:27.580584 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566123 4455 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:52:27.580584 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566127 4455 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:52:27.580584 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566131 4455 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:52:27.580584 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566136 4455 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:52:27.580584 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566140 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:52:27.580584 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566144 4455 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:52:27.580584 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566148 4455 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:52:27.580584 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566153 4455 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:52:27.580584 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566157 4455 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:52:27.580584 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566161 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:52:27.580584 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566166 4455 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:52:27.581072 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566171 4455 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:52:27.581072 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566175 4455 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:52:27.581072 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566179 4455 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:52:27.581072 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566183 4455 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:52:27.581072 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566187 4455 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:52:27.581072 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566193 4455 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:52:27.581072 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566197 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:52:27.581072 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566201 4455 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:52:27.581072 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566205 4455 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:52:27.581072 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566210 4455 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:52:27.581072 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566214 4455 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:52:27.581072 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566218 4455 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:52:27.581072 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566222 4455 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:52:27.581072 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566226 4455 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:52:27.581072 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566230 4455 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:52:27.581072 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566234 4455 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:52:27.581072 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566238 4455 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:52:27.581072 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566243 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:52:27.581072 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566247 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:52:27.581072 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566251 4455 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:52:27.581606 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566255 4455 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:52:27.581606 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566259 4455 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:52:27.581606 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566263 4455 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:52:27.581606 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566267 4455 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:52:27.581606 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566271 4455 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:52:27.581606 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566275 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:52:27.581606 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566279 4455 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:52:27.581606 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.566337 4455 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:52:27.581606 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566561 4455 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:52:27.581606 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566569 4455 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:52:27.581606 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566574 4455 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:52:27.581606 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566579 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:52:27.581606 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566583 4455 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:52:27.581606 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566588 4455 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:52:27.581606 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566591 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:52:27.581606 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566595 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:52:27.581980 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566599 4455 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:52:27.581980 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566603 4455 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:52:27.581980 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566608 4455 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:52:27.581980 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566612 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:52:27.581980 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566616 4455 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:52:27.581980 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566620 4455 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:52:27.581980 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566625 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:52:27.581980 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566629 4455 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:52:27.581980 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566634 4455 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:52:27.581980 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566638 4455 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:52:27.581980 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566642 4455 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:52:27.581980 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566646 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:52:27.581980 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566652 4455 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:52:27.581980 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566656 4455 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:52:27.581980 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566661 4455 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:52:27.581980 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566665 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:52:27.581980 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566668 4455 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:52:27.581980 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566673 4455 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:52:27.581980 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566679 4455 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:52:27.582442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566685 4455 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:52:27.582442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566691 4455 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:52:27.582442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566699 4455 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:52:27.582442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566706 4455 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:52:27.582442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566710 4455 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:52:27.582442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566715 4455 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:52:27.582442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566719 4455 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:52:27.582442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566724 4455 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:52:27.582442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566728 4455 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:52:27.582442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566732 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:52:27.582442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566736 4455 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:52:27.582442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566741 4455 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:52:27.582442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566745 4455 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:52:27.582442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566750 4455 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:52:27.582442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566754 4455 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:52:27.582442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566758 4455 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:52:27.582442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566768 4455 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:52:27.582442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566772 4455 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:52:27.582442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566777 4455 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:52:27.582442 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566781 4455 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:52:27.582923 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566786 4455 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:52:27.582923 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566790 4455 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:52:27.582923 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566794 4455 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:52:27.582923 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566798 4455 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:52:27.582923 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566802 4455 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:52:27.582923 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566806 4455 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:52:27.582923 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566811 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:52:27.582923 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566815 4455 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:52:27.582923 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566819 4455 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:52:27.582923 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566823 4455 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:52:27.582923 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566827 4455 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:52:27.582923 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566831 4455 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:52:27.582923 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566836 4455 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:52:27.582923 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566840 4455 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:52:27.582923 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566844 4455 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:52:27.582923 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566850 4455 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:52:27.582923 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566855 4455 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:52:27.582923 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566859 4455 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:52:27.582923 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566863 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:52:27.583405 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566868 4455 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:52:27.583405 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566872 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:52:27.583405 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566876 4455 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:52:27.583405 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566880 4455 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:52:27.583405 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566885 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:52:27.583405 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566889 4455 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:52:27.583405 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566893 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:52:27.583405 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566898 4455 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:52:27.583405 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566902 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:52:27.583405 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566906 4455 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:52:27.583405 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566912 4455 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:52:27.583405 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566917 4455 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:52:27.583405 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566921 4455 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:52:27.583405 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566925 4455 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:52:27.583405 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566929 4455 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:52:27.583405 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566933 4455 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:52:27.583405 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566938 4455 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:52:27.583405 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566942 4455 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:52:27.583405 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566946 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:52:27.583405 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566951 4455 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:52:27.583890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566955 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:52:27.583890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566959 4455 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:52:27.583890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566963 4455 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:52:27.583890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566967 4455 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:52:27.583890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566971 4455 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:52:27.583890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566975 4455 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:52:27.583890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566980 4455 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:52:27.583890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566984 4455 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:52:27.583890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566989 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:52:27.583890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566994 4455 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:52:27.583890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.566998 4455 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:52:27.583890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567002 4455 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:52:27.583890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567006 4455 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:52:27.583890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567010 4455 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:52:27.583890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567014 4455 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:52:27.583890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567018 4455 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:52:27.583890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567022 4455 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:52:27.583890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567027 4455 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:52:27.583890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567031 4455 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:52:27.583890 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567043 4455 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:52:27.584381 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.567085 4455 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:52:27.584381 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567139 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:52:27.584381 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567145 4455 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:52:27.584381 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567151 4455 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:52:27.584381 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567156 4455 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:52:27.584381 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567160 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:52:27.584381 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567164 4455 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:52:27.584381 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567168 4455 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:52:27.584381 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567173 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:52:27.584381 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567177 4455 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:52:27.584381 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567182 4455 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:52:27.584381 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567186 4455 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:52:27.584381 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567190 4455 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:52:27.584381 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567194 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:52:27.584381 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567199 4455 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:52:27.584381 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567203 4455 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:52:27.584777 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567208 4455 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:52:27.584777 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567212 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:52:27.584777 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567216 4455 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:52:27.584777 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567221 4455 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:52:27.584777 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567227 4455 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:52:27.584777 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567235 4455 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:52:27.584777 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567241 4455 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:52:27.584777 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567248 4455 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:52:27.584777 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567254 4455 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:52:27.584777 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567259 4455 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:52:27.584777 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567263 4455 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:52:27.584777 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567268 4455 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:52:27.584777 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567272 4455 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:52:27.584777 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567276 4455 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:52:27.584777 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567280 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:52:27.584777 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567284 4455 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:52:27.584777 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567289 4455 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:52:27.584777 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567293 4455 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:52:27.584777 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567298 4455 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:52:27.585214 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567302 4455 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:52:27.585214 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567307 4455 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:52:27.585214 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567312 4455 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:52:27.585214 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567316 4455 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:52:27.585214 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567337 4455 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:52:27.585214 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567341 4455 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:52:27.585214 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567346 4455 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:52:27.585214 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567350 4455 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:52:27.585214 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567354 4455 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:52:27.585214 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567358 4455 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:52:27.585214 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567362 4455 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:52:27.585214 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567366 4455 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:52:27.585214 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567370 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:52:27.585214 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567374 4455 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:52:27.585214 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567378 4455 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:52:27.585214 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567383 4455 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:52:27.585214 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567387 4455 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:52:27.585214 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567391 4455 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:52:27.585214 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567397 4455 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:52:27.585214 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567402 4455 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:52:27.585933 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567406 4455 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:52:27.585933 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567410 4455 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:52:27.585933 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567414 4455 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:52:27.585933 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567418 4455 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:52:27.585933 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567423 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:52:27.585933 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567429 4455 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:52:27.585933 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567433 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:52:27.585933 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567437 4455 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:52:27.585933 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567441 4455 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:52:27.585933 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567445 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:52:27.585933 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567450 4455 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:52:27.585933 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567454 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:52:27.585933 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567458 4455 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:52:27.585933 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567462 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:52:27.585933 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567468 4455 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:52:27.585933 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567473 4455 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:52:27.585933 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567478 4455 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:52:27.585933 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567482 4455 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:52:27.585933 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567486 4455 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:52:27.586842 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567490 4455 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:52:27.586842 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567495 4455 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:52:27.586842 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567499 4455 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:52:27.586842 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567505 4455 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:52:27.586842 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567509 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:52:27.586842 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567513 4455 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:52:27.586842 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567518 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:52:27.586842 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567522 4455 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:52:27.586842 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567526 4455 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:52:27.586842 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567530 4455 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:52:27.586842 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567534 4455 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:52:27.586842 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567541 4455 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:52:27.586842 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567545 4455 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:52:27.586842 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567549 4455 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:52:27.586842 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567554 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:52:27.586842 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567558 4455 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:52:27.586842 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567562 4455 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:52:27.586842 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567566 4455 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:52:27.586842 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567570 4455 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:52:27.586842 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567574 4455 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:52:27.587814 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567578 4455 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:52:27.587814 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567582 4455 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:52:27.587814 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567586 4455 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:52:27.587814 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567591 4455 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:52:27.587814 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567595 4455 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:52:27.587814 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567599 4455 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:52:27.587814 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567603 4455 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:52:27.587814 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567607 4455 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:52:27.587814 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567612 4455 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:52:27.587814 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567616 4455 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:52:27.587814 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567620 4455 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:52:27.587814 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567626 4455 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:52:27.587814 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:27.567630 4455 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:52:27.587814 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.567713 4455 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 17:52:27.587814 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.569242 4455 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 17:52:27.587814 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.569401 4455 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 17:52:27.587814 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.569679 4455 server.go:1014] "Starting client certificate rotation" Oct 30 17:52:27.587814 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.569807 4455 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 17:52:27.588753 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.569857 4455 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 17:46:47 +0000 UTC" deadline="2028-04-07 22:46:38.087994942 +0000 UTC" Oct 30 17:52:27.588753 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.569893 4455 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12604h54m10.518104539s" Oct 30 17:52:27.588753 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.570416 4455 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:52:27.588753 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.572113 4455 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:52:27.588753 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.573780 4455 log.go:25] "Validated CRI v1 runtime API" Oct 30 17:52:27.588753 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.575212 4455 log.go:25] "Validated CRI v1 image API" Oct 30 17:52:27.588753 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.575933 4455 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 17:52:27.588753 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.577360 4455 fs.go:135] Filesystem UUIDs: map[6058e3e4-674f-4c8d-a77f-eb4f26f262d9:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ed7cf5a5-e0a5-4f68-9534-cfa0f67ed3bb:/dev/nvme0n1p4] Oct 30 17:52:27.589132 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.577379 4455 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0d5746ebfc1c8717d8e753c111dc4d7a251c1a24123463dcb3a9b263b11cfc49/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0d5746ebfc1c8717d8e753c111dc4d7a251c1a24123463dcb3a9b263b11cfc49/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/154701d0f49a1cff69b8eb04b125f9cdc160ef78aeafb7ac1d6a9d77708ec895/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/154701d0f49a1cff69b8eb04b125f9cdc160ef78aeafb7ac1d6a9d77708ec895/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3f7389d17e5cf41130e288aac5921695740d030d2937f61a40db6d65ab9f8dc6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3f7389d17e5cf41130e288aac5921695740d030d2937f61a40db6d65ab9f8dc6/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/55e996343a802cdda6b8be5b128355933f273d87b7b858dbcc81d9d8c53a27e0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/55e996343a802cdda6b8be5b128355933f273d87b7b858dbcc81d9d8c53a27e0/userdata/shm major:0 minor:215 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/890471f8caa56baae6e88be363425b85cf5b83af0837023213d084bbcc0441ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/890471f8caa56baae6e88be363425b85cf5b83af0837023213d084bbcc0441ca/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/90dd88d0b6a648f009bdfc07e21ba45f5aafa9255dfd19e15012a0509f9d72a0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/90dd88d0b6a648f009bdfc07e21ba45f5aafa9255dfd19e15012a0509f9d72a0/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a3663bec9c925c88b65e04235931d37e954e7c5375bd99f92a6eae78fe533e71/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a3663bec9c925c88b65e04235931d37e954e7c5375bd99f92a6eae78fe533e71/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b286dc59e270f5f38fd27ef90603fbbd38a397f5db1f21bfa70e2e9caa3fe25e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b286dc59e270f5f38fd27ef90603fbbd38a397f5db1f21bfa70e2e9caa3fe25e/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b46c2a539b3aedbbe6198927c9b223a42e6a8786a6c028393a61d883c75ae4d0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b46c2a539b3aedbbe6198927c9b223a42e6a8786a6c028393a61d883c75ae4d0/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b78aa875ef7cb538c5307aa35e52712ad1b773d813d0833e51e5d96437354016/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b78aa875ef7cb538c5307aa35e52712ad1b773d813d0833e51e5d96437354016/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bcd3d19f28b9e870d6cafcb6ce6918d8f2537f72f27eacb23269e70d9c4b50cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bcd3d19f28b9e870d6cafcb6ce6918d8f2537f72f27eacb23269e70d9c4b50cf/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd497f78610265355b84c9770790898f4b6dace5069c46cbb68eebf4ba9c48ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd497f78610265355b84c9770790898f4b6dace5069c46cbb68eebf4ba9c48ce/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf97c2e0988205e99b5e8a6c36b3f10a7aacf72bca36e0643596739b693d573e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf97c2e0988205e99b5e8a6c36b3f10a7aacf72bca36e0643596739b693d573e/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d38f96c68a13e957832eb9e4bafbfc7db916f074f29e4a8bfd8efaf32a605148/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d38f96c68a13e957832eb9e4bafbfc7db916f074f29e4a8bfd8efaf32a605148/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13521cae-dc8a-4670-9321-61ad6ab3fd4d/volumes/kubernetes.io~projected/kube-api-access-5vbfw:{mountpoint:/var/lib/kubelet/pods/13521cae-dc8a-4670-9321-61ad6ab3fd4d/volumes/kubernetes.io~projected/kube-api-access-5vbfw major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fb157ba-42c2-4efc-a76b-6863fc987fb6/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/1fb157ba-42c2-4efc-a76b-6863fc987fb6/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b7db1b0-8abd-47fb-9df8-7caab8adfed6/volumes/kubernetes.io~projected/kube-api-access-l864g:{mountpoint:/var/lib/kubelet/pods/3b7db1b0-8abd-47fb-9df8-7caab8adfed6/volumes/kubernetes.io~projected/kube-api-access-l864g major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60c3d534-02fc-4665-ada2-e56fba531b87/volumes/kubernetes.io~projected/kube-api-access-mww6s:{mountpoint:/var/lib/kubelet/pods/60c3d534-02fc-4665-ada2-e56fba531b87/volumes/kubernetes.io~projected/kube-api-access-mww6s major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60c3d534-02fc-4665-ada2-e56fba531b87/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/60c3d534-02fc-4665-ada2-e56fba531b87/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63b2e2e0-805f-4197-81d5-6eafe3831ded/volumes/kubernetes.io~projected/kube-api-access-q25cz:{mountpoint:/var/lib/kubelet/pods/63b2e2e0-805f-4197-81d5-6eafe3831ded/volumes/kubernetes.io~projected/kube-api-access-q25cz major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73dbe36c-9ccb-47d6-b11e-afc35b87008e/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/73dbe36c-9ccb-47d6-b11e-afc35b87008e/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75cac6ab-2f87-4813-80a1-cc6efb29d187/volumes/kubernetes.io~projected/kube-api-access-pbjzl:{mountpoint:/var/lib/kubelet/pods/75cac6ab-2f87-4813-80a1-cc6efb29d187/volumes/kubernetes.io~projected/kube-api-access-pbjzl major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f348173-6e5e-404a-9126-7aa648f7eb01/volumes/kubernetes.io~projected/kube-api-access-5rhhq:{mountpoint:/var/lib/kubelet/pods/8f348173-6e5e-404a-9126-7aa648f7eb01/volumes/kubernetes.io~projected/kube-api-access-5rhhq major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b38a6e02-c51f-455b-b5bd-25ca428cddb2/volumes/kubernetes.io~projected/kube-api-access-n7lk6:{mountpoint:/var/lib/kubelet/pods/b38a6e02-c51f-455b-b5bd-25ca428cddb2/volumes/kubernetes.io~projected/kube-api-access-n7lk6 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c126d55e-cb37-4070-904d-47f76d781751/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/c126d55e-cb37-4070-904d-47f76d781751/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c126d55e-cb37-4070-904d-47f76d781751/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/c126d55e-cb37-4070-904d-47f76d781751/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c126d55e-cb37-4070-904d-47f76d781751/volumes/kubernetes.io~projected/kube-api-access-sbw76:{mountpoint:/var/lib/kubelet/pods/c126d55e-cb37-4070-904d-47f76d781751/volumes/kubernetes.io~projected/kube-api-access-sbw76 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9e23597-79e7-4a28-ab52-fc8b343226ab/volumes/kubernetes.io~projected/kube-api-access-lnxpn:{mountpoint:/var/lib/kubelet/pods/c9e23597-79e7-4a28-ab52-fc8b343226ab/volumes/kubernetes.io~projected/kube-api-access-lnxpn major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9e23597-79e7-4a28-ab52-fc8b343226ab/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/c9e23597-79e7-4a28-ab52-fc8b343226ab/volumes/kubernetes.io~secret/metrics-tls major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb68da46-11b6-48f4-b302-3ad63d93f82f/volumes/kubernetes.io~projected/kube-api-access-mppvt:{mountpoint:/var/lib/kubelet/pods/cb68da46-11b6-48f4-b302-3ad63d93f82f/volumes/kubernetes.io~projected/kube-api-access-mppvt major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbda8c42-0f24-41a3-8a5f-727a4094d021/volumes/kubernetes.io~projected/kube-api-access-ccflp:{mountpoint:/var/lib/kubelet/pods/cbda8c42-0f24-41a3-8a5f-727a4094d021/volumes/kubernetes.io~projected/kube-api-access-ccflp major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1397e4e-e615-4954-8486-af559b65687e/volumes/kubernetes.io~projected/kube-api-access-b2d2s:{mountpoint:/var/lib/kubelet/pods/e1397e4e-e615-4954-8486-af559b65687e/volumes/kubernetes.io~projected/kube-api-access-b2d2s major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f983c48d-08c4-41e2-af03-7c36cf651faa/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f983c48d-08c4-41e2-af03-7c36cf651faa/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f983c48d-08c4-41e2-af03-7c36cf651faa/volumes/kubernetes.io~projected/kube-api-access-ljxjz:{mountpoint:/var/lib/kubelet/pods/f983c48d-08c4-41e2-af03-7c36cf651faa/volumes/kubernetes.io~projected/kube-api-access-ljxjz major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f983c48d-08c4-41e2-af03-7c36cf651faa/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f983c48d-08c4-41e2-af03-7c36cf651faa/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/988bc501e2059abeaf78ac5249b1bb5b181af871c31ca32341abde11f50aeee3/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/5a851acde2dd19651209e7f7921cc5e286c05a86a8f30bf5325bfb877c864431/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/2824ed40fa15f0db8aa78a49a1abe0d8cd193c2f8ac692572a57ad045a8567ea/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/2212af6c9686f2070df8e63b159316d47e105af2a8964ef5fe72a77d4e263ce8/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/0783542266e1ed52668571ee22a1947fd9bf4b1c9e7a31df3ec6d954a5838b94/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/da9cd06517805e0d100b4208cb24e3815aff52fa5058d8a5bc32861d161623e5/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/1e01073166607d22ac260d25c5aeb9c07aeff38b3fe7862326ebbe0bd35f8dd1/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/29656508b2fd211db5aaa56ee644793f8edf511cd286289a8d782a560d1820ef/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/4b24018cb43606e29dc5d532800d587992657972150842003d73a379620010e1/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/998dd67974384a7a658c06a4f65d653523466e7eb3efe80021d622c4f583738a/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/b8d14c78d2ebb5017122c9c3424b468549d351548141d26e716e43f4f2fef45c/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-145:{mountpoint:/var/lib/containers/storage/overlay/76c2a84563a02af8bac2410727b42eff7b28931c5d36c53de52165ef438d06ea/merged major:0 minor:145 fsType:overlay blockSize:0} overlay_0-148:{mountpoint:/var/lib/containers/storage/overlay/d1588f8e7a0c3e4d49a17022b842dbf827bc816cd21bca70ca6cd355aa01bc26/merged major:0 minor:148 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/48a0538daf5255e6d0bfcfa71d7f0fa316043ea8fbd27dd00b941c19c1037aeb/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/c844d7d5ae23bb9dafb879da5302bef0a497a9cc25864a18016c0828f9f0edef/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/d797fb42df0be99ac8f028535a6772e80cfa21a533a2ce651fde6eef451ebeeb/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/d1f522d777dad5c9fd9bbb67ff8392cff6feb27ca5f979d1ee7129421b4cf226/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/f00b2f270bddf24a37258eb215f6eda919ffa96fc9bca17e0e7fdc2ce2cfd6a8/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/7b0aeaa463565c879d3300673bf9b741b95629ddf5678715678dcfea55c8f998/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-184:{mountpoint:/var/lib/containers/storage/overlay/e45cf2748d29ee24ccc594e61a882cb8af41a0c3a104d530429934810c0e9f33/merged major:0 minor:184 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/45b34177ae3ab2123da65fe9a9366f2de210152e38a7d8834640ca20bd7f3e52/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/36cf234b3b9cb86912255fd7101985b7752a2a078955bd9fa1d21a341ecfd012/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/906416aaca5d54f597df852b60da38839ed17af31a77b7991a01665eabcd031b/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/3c999517f1112276e7ba3dc0f3b620805b0f882ce4a4285ef49f85b5023c9dc8/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/3d72a4bdb601a2b591c334e0b5b557b54f6cc4bbdf9c315d57b57351b8720c7f/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/a9d83c998a55ffae7a2b93b2175f1beff737f9352e91f3468cc08bf436fdf0df/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/663668d333e22ea8689fc2ebb60b013360c9dc62e5d5e6dcc6003671f1468c2a/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/bb4488bff2efa0dd5062ad2a2f4ac7a129d2e114ef443beeb538c4d0a117a8e7/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/fd72c424cad15fa41fc812198f9e356c0592205225e0e20ebe9292af0dbfe69a/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/7c5af7a0dc4e17f6748d16ad9234fa5c4803668d52c464ccd8fe5fd1ee14a9f9/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/0b77a43ca17efd76ad9ace08226bda01c48f942be5e4f0f706350c710196bbae/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/a8e09ef3246c457a9566259b394a5675136f658e869ff9c321f487da5b250f93/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/062c2f217c853bc4916d3b32b2338c6dccbb7d89679748996197a6383c1a5b11/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/86bc1f125e7a42b8d7c3b32bdfc88ce5bee38e0d2a69b1189e52c23b39dd87c2/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/7b5c5f018e67097a1aaf83b168c5d7c3be2014cfd42aa5664e801126edb375e4/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/c1bc41d221cef42d939a606b8ef023c14614bbaa03c67acb133b855002203579/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/79ea3d6c05c3d5fdf74baff0aa01d1c790126c6fc04e99250c3e6f3b6c380ec5/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/50c0973f026b89a26a2bf0d8759b4518c3a01cbda553fd3e101c372739b564a7/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/0bae9c69a16ec666e55819c9a09eb3be235ec8011c689f7bd23e6e99867825d6/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/c88dc7175a4f96cea15d8d857a212957d52dc6507c86aea8e043807e916db2b6/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 30 17:52:27.589369 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.584366 4455 manager.go:217] Machine: {Timestamp:2026-10-30 17:52:27.583880067 +0000 UTC m=+0.071517092 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099186 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2869ed0b07fc42e7e792d2d96458d6 SystemUUID:ec2869ed-0b07-fc42-e7e7-92d2d96458d6 BootID:9f1dd574-157a-4a3c-be46-bc2b290da3f3 Filesystems:[{Device:/var/lib/kubelet/pods/cb68da46-11b6-48f4-b302-3ad63d93f82f/volumes/kubernetes.io~projected/kube-api-access-mppvt DeviceMajor:0 DeviceMinor:214 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bcd3d19f28b9e870d6cafcb6ce6918d8f2537f72f27eacb23269e70d9c4b50cf/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/cbda8c42-0f24-41a3-8a5f-727a4094d021/volumes/kubernetes.io~projected/kube-api-access-ccflp DeviceMajor:0 DeviceMinor:59 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/c126d55e-cb37-4070-904d-47f76d781751/volumes/kubernetes.io~projected/kube-api-access-sbw76 DeviceMajor:0 DeviceMinor:62 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3f7389d17e5cf41130e288aac5921695740d030d2937f61a40db6d65ab9f8dc6/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-184 DeviceMajor:0 DeviceMinor:184 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b286dc59e270f5f38fd27ef90603fbbd38a397f5db1f21bfa70e2e9caa3fe25e/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f983c48d-08c4-41e2-af03-7c36cf651faa/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/13521cae-dc8a-4670-9321-61ad6ab3fd4d/volumes/kubernetes.io~projected/kube-api-access-5vbfw DeviceMajor:0 DeviceMinor:61 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e1397e4e-e615-4954-8486-af559b65687e/volumes/kubernetes.io~projected/kube-api-access-b2d2s DeviceMajor:0 DeviceMinor:67 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/60c3d534-02fc-4665-ada2-e56fba531b87/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/90dd88d0b6a648f009bdfc07e21ba45f5aafa9255dfd19e15012a0509f9d72a0/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/63b2e2e0-805f-4197-81d5-6eafe3831ded/volumes/kubernetes.io~projected/kube-api-access-q25cz DeviceMajor:0 DeviceMinor:66 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-145 DeviceMajor:0 DeviceMinor:145 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/890471f8caa56baae6e88be363425b85cf5b83af0837023213d084bbcc0441ca/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a3663bec9c925c88b65e04235931d37e954e7c5375bd99f92a6eae78fe533e71/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f983c48d-08c4-41e2-af03-7c36cf651faa/volumes/kubernetes.io~projected/kube-api-access-ljxjz DeviceMajor:0 DeviceMinor:64 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b38a6e02-c51f-455b-b5bd-25ca428cddb2/volumes/kubernetes.io~projected/kube-api-access-n7lk6 DeviceMajor:0 DeviceMinor:201 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/1fb157ba-42c2-4efc-a76b-6863fc987fb6/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:209 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/75cac6ab-2f87-4813-80a1-cc6efb29d187/volumes/kubernetes.io~projected/kube-api-access-pbjzl DeviceMajor:0 DeviceMinor:65 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/60c3d534-02fc-4665-ada2-e56fba531b87/volumes/kubernetes.io~projected/kube-api-access-mww6s DeviceMajor:0 DeviceMinor:197 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/55e996343a802cdda6b8be5b128355933f273d87b7b858dbcc81d9d8c53a27e0/userdata/shm DeviceMajor:0 DeviceMinor:215 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/c126d55e-cb37-4070-904d-47f76d781751/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bd497f78610265355b84c9770790898f4b6dace5069c46cbb68eebf4ba9c48ce/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0d5746ebfc1c8717d8e753c111dc4d7a251c1a24123463dcb3a9b263b11cfc49/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/3b7db1b0-8abd-47fb-9df8-7caab8adfed6/volumes/kubernetes.io~projected/kube-api-access-l864g DeviceMajor:0 DeviceMinor:63 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f983c48d-08c4-41e2-af03-7c36cf651faa/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8f348173-6e5e-404a-9126-7aa648f7eb01/volumes/kubernetes.io~projected/kube-api-access-5rhhq DeviceMajor:0 DeviceMinor:60 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/154701d0f49a1cff69b8eb04b125f9cdc160ef78aeafb7ac1d6a9d77708ec895/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf97c2e0988205e99b5e8a6c36b3f10a7aacf72bca36e0643596739b693d573e/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d38f96c68a13e957832eb9e4bafbfc7db916f074f29e4a8bfd8efaf32a605148/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c9e23597-79e7-4a28-ab52-fc8b343226ab/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:223 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-148 DeviceMajor:0 DeviceMinor:148 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c126d55e-cb37-4070-904d-47f76d781751/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/73dbe36c-9ccb-47d6-b11e-afc35b87008e/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b78aa875ef7cb538c5307aa35e52712ad1b773d813d0833e51e5d96437354016/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b46c2a539b3aedbbe6198927c9b223a42e6a8786a6c028393a61d883c75ae4d0/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c9e23597-79e7-4a28-ab52-fc8b343226ab/volumes/kubernetes.io~projected/kube-api-access-lnxpn DeviceMajor:0 DeviceMinor:202 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0d5746ebfc1c871 MacAddress:aa:a4:ea:19:17:ae Speed:10000 Mtu:8901} {Name:55e996343a802cd MacAddress:1e:94:74:87:85:84 Speed:10000 Mtu:8901} {Name:b78aa875ef7cb53 MacAddress:b2:9b:1a:7b:3e:c2 Speed:10000 Mtu:8901} {Name:bcd3d19f28b9e87 MacAddress:66:4d:d7:15:8b:e0 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d6:b9:1c:57 Speed:0 Mtu:9001} {Name:br-int MacAddress:b2:aa:55:6a:72:03 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:d6:b9:1c:57 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:e2:dd:bb:e0:67:ed Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:06:bb:4b:53:1b:e7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 17:52:27.589369 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.584609 4455 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 17:52:27.589369 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.584651 4455 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 17:52:27.589369 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.584885 4455 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 17:52:27.589369 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.584907 4455 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-84.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 17:52:27.589369 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.585035 4455 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 17:52:27.589369 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.585043 4455 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 17:52:27.589369 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.585052 4455 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:52:27.589369 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.585065 4455 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:52:27.589369 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.585158 4455 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 17:52:27.589369 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.585333 4455 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 17:52:27.589369 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.585446 4455 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 17:52:27.589369 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.585479 4455 kubelet.go:493] "Attempting to sync node with API server" Oct 30 17:52:27.589369 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.585488 4455 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 17:52:27.589369 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.585498 4455 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 17:52:27.589369 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.585507 4455 kubelet.go:405] "Adding apiserver pod source" Oct 30 17:52:27.589369 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.585515 4455 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 17:52:27.589369 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.586093 4455 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 17:52:27.589369 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.586105 4455 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:52:27.589369 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.586671 4455 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 17:52:27.589369 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.589002 4455 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 17:52:27.589369 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.589014 4455 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:52:27.589369 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.589113 4455 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 17:52:27.589369 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.589337 4455 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 17:52:27.589369 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.589368 4455 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 17:52:27.590662 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.589463 4455 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 17:52:27.590662 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.589475 4455 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 17:52:27.590662 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.589481 4455 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 17:52:27.590662 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.589489 4455 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 17:52:27.590662 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.589497 4455 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 17:52:27.590662 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.589502 4455 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 17:52:27.590662 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.589511 4455 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 17:52:27.590662 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.589517 4455 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 17:52:27.590662 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.589527 4455 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 17:52:27.590662 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.589555 4455 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 17:52:27.590662 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.589573 4455 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 17:52:27.590662 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.589583 4455 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 17:52:27.590662 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.589606 4455 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 17:52:27.590662 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.589611 4455 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 17:52:27.593309 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.593296 4455 server.go:1263] "Started kubelet" Oct 30 17:52:27.593418 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.593390 4455 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 17:52:27.593668 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.593542 4455 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 17:52:27.594930 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.593802 4455 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 17:52:27.594930 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.594305 4455 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 17:52:27.593916 ip-10-0-130-84 systemd[1]: Started Kubernetes Kubelet. Oct 30 17:52:27.599187 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.597219 4455 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 17:52:27.599187 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.597547 4455 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 17:52:27.599187 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.597562 4455 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 17:52:27.599187 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.597630 4455 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 17:52:27.599187 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.597577 4455 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 17:52:27.599187 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.597802 4455 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:46:49 +0000 UTC" deadline="2028-06-06 21:15:37.436247206 +0000 UTC" Oct 30 17:52:27.599187 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.597862 4455 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14043h23m9.838389596s" Oct 30 17:52:27.599187 ip-10-0-130-84 kubenswrapper[4455]: E1030 17:52:27.598199 4455 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-84.ec2.internal\" not found" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600150 4455 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600175 4455 factory.go:55] Registering systemd factory Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600187 4455 factory.go:223] Registration of the systemd container factory successfully Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600201 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63b2e2e0-805f-4197-81d5-6eafe3831ded" volumeName="kubernetes.io/projected/63b2e2e0-805f-4197-81d5-6eafe3831ded-kube-api-access-q25cz" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600240 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb68da46-11b6-48f4-b302-3ad63d93f82f" volumeName="kubernetes.io/projected/cb68da46-11b6-48f4-b302-3ad63d93f82f-kube-api-access-mppvt" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600256 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbda8c42-0f24-41a3-8a5f-727a4094d021" volumeName="kubernetes.io/configmap/cbda8c42-0f24-41a3-8a5f-727a4094d021-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600275 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f983c48d-08c4-41e2-af03-7c36cf651faa" volumeName="kubernetes.io/projected/f983c48d-08c4-41e2-af03-7c36cf651faa-kube-api-access-ljxjz" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600290 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1397e4e-e615-4954-8486-af559b65687e" volumeName="kubernetes.io/projected/e1397e4e-e615-4954-8486-af559b65687e-kube-api-access-b2d2s" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600350 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f983c48d-08c4-41e2-af03-7c36cf651faa" volumeName="kubernetes.io/configmap/f983c48d-08c4-41e2-af03-7c36cf651faa-ovnkube-script-lib" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600365 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f983c48d-08c4-41e2-af03-7c36cf651faa" volumeName="kubernetes.io/configmap/f983c48d-08c4-41e2-af03-7c36cf651faa-env-overrides" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600377 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f983c48d-08c4-41e2-af03-7c36cf651faa" volumeName="kubernetes.io/configmap/f983c48d-08c4-41e2-af03-7c36cf651faa-ovnkube-config" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600400 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9e23597-79e7-4a28-ab52-fc8b343226ab" volumeName="kubernetes.io/projected/c9e23597-79e7-4a28-ab52-fc8b343226ab-kube-api-access-lnxpn" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600414 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbda8c42-0f24-41a3-8a5f-727a4094d021" volumeName="kubernetes.io/configmap/cbda8c42-0f24-41a3-8a5f-727a4094d021-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600431 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13521cae-dc8a-4670-9321-61ad6ab3fd4d" volumeName="kubernetes.io/projected/13521cae-dc8a-4670-9321-61ad6ab3fd4d-kube-api-access-5vbfw" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600512 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b7db1b0-8abd-47fb-9df8-7caab8adfed6" volumeName="kubernetes.io/projected/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-kube-api-access-l864g" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600528 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60c3d534-02fc-4665-ada2-e56fba531b87" volumeName="kubernetes.io/secret/60c3d534-02fc-4665-ada2-e56fba531b87-cert" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600545 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63b2e2e0-805f-4197-81d5-6eafe3831ded" volumeName="kubernetes.io/configmap/63b2e2e0-805f-4197-81d5-6eafe3831ded-multus-daemon-config" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600586 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63b2e2e0-805f-4197-81d5-6eafe3831ded" volumeName="kubernetes.io/configmap/63b2e2e0-805f-4197-81d5-6eafe3831ded-cni-binary-copy" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600601 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f348173-6e5e-404a-9126-7aa648f7eb01" volumeName="kubernetes.io/projected/8f348173-6e5e-404a-9126-7aa648f7eb01-kube-api-access-5rhhq" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600615 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75cac6ab-2f87-4813-80a1-cc6efb29d187" volumeName="kubernetes.io/projected/75cac6ab-2f87-4813-80a1-cc6efb29d187-kube-api-access-pbjzl" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600643 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b38a6e02-c51f-455b-b5bd-25ca428cddb2" volumeName="kubernetes.io/configmap/b38a6e02-c51f-455b-b5bd-25ca428cddb2-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600658 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b38a6e02-c51f-455b-b5bd-25ca428cddb2" volumeName="kubernetes.io/empty-dir/b38a6e02-c51f-455b-b5bd-25ca428cddb2-data-volume" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600678 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9e23597-79e7-4a28-ab52-fc8b343226ab" volumeName="kubernetes.io/configmap/c9e23597-79e7-4a28-ab52-fc8b343226ab-config-volume" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600691 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbda8c42-0f24-41a3-8a5f-727a4094d021" volumeName="kubernetes.io/projected/cbda8c42-0f24-41a3-8a5f-727a4094d021-kube-api-access-ccflp" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600705 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1397e4e-e615-4954-8486-af559b65687e" volumeName="kubernetes.io/empty-dir/e1397e4e-e615-4954-8486-af559b65687e-tmp-dir" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600723 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f983c48d-08c4-41e2-af03-7c36cf651faa" volumeName="kubernetes.io/secret/f983c48d-08c4-41e2-af03-7c36cf651faa-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600736 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f348173-6e5e-404a-9126-7aa648f7eb01" volumeName="kubernetes.io/configmap/8f348173-6e5e-404a-9126-7aa648f7eb01-iptables-alerter-script" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600757 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b38a6e02-c51f-455b-b5bd-25ca428cddb2" volumeName="kubernetes.io/projected/b38a6e02-c51f-455b-b5bd-25ca428cddb2-kube-api-access-n7lk6" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600769 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c126d55e-cb37-4070-904d-47f76d781751" volumeName="kubernetes.io/empty-dir/c126d55e-cb37-4070-904d-47f76d781751-tmp" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600784 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbda8c42-0f24-41a3-8a5f-727a4094d021" volumeName="kubernetes.io/configmap/cbda8c42-0f24-41a3-8a5f-727a4094d021-cni-binary-copy" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600813 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73dbe36c-9ccb-47d6-b11e-afc35b87008e" volumeName="kubernetes.io/configmap/73dbe36c-9ccb-47d6-b11e-afc35b87008e-konnectivity-ca" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600839 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73dbe36c-9ccb-47d6-b11e-afc35b87008e" volumeName="kubernetes.io/secret/73dbe36c-9ccb-47d6-b11e-afc35b87008e-agent-certs" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600875 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fb157ba-42c2-4efc-a76b-6863fc987fb6" volumeName="kubernetes.io/secret/1fb157ba-42c2-4efc-a76b-6863fc987fb6-original-pull-secret" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600891 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c126d55e-cb37-4070-904d-47f76d781751" volumeName="kubernetes.io/projected/c126d55e-cb37-4070-904d-47f76d781751-kube-api-access-sbw76" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600937 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9e23597-79e7-4a28-ab52-fc8b343226ab" volumeName="kubernetes.io/secret/c9e23597-79e7-4a28-ab52-fc8b343226ab-metrics-tls" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600952 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60c3d534-02fc-4665-ada2-e56fba531b87" volumeName="kubernetes.io/projected/60c3d534-02fc-4665-ada2-e56fba531b87-kube-api-access-mww6s" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600980 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75cac6ab-2f87-4813-80a1-cc6efb29d187" volumeName="kubernetes.io/configmap/75cac6ab-2f87-4813-80a1-cc6efb29d187-serviceca" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.600993 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c126d55e-cb37-4070-904d-47f76d781751" volumeName="kubernetes.io/empty-dir/c126d55e-cb37-4070-904d-47f76d781751-etc-tuned" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.601007 4455 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9e23597-79e7-4a28-ab52-fc8b343226ab" volumeName="kubernetes.io/empty-dir/c9e23597-79e7-4a28-ab52-fc8b343226ab-tmp-dir" seLinuxMountContext="" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.601022 4455 reconstruct.go:97] "Volume reconstruction finished" Oct 30 17:52:27.601625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.601029 4455 reconciler.go:29] "Reconciler: start to sync state" Oct 30 17:52:27.603283 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.602668 4455 factory.go:153] Registering CRI-O factory Oct 30 17:52:27.603283 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.602684 4455 factory.go:223] Registration of the crio container factory successfully Oct 30 17:52:27.603283 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.602704 4455 factory.go:103] Registering Raw factory Oct 30 17:52:27.603283 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.602718 4455 manager.go:1199] Started watching for new ooms in manager Oct 30 17:52:27.603283 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.603122 4455 manager.go:319] Starting recovery of all containers Oct 30 17:52:27.603781 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.603755 4455 server.go:317] "Adding debug handlers to kubelet server" Oct 30 17:52:27.624617 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.624570 4455 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 17:52:27.625920 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.625897 4455 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 17:52:27.626052 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.625926 4455 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 17:52:27.626052 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.625960 4455 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 17:52:27.626052 ip-10-0-130-84 kubenswrapper[4455]: E1030 17:52:27.626000 4455 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 17:52:27.643488 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.643423 4455 generic.go:356] "Generic (PLEG): container finished" podID="5ed893d4ed91b2f423e686d98f2bd57e" containerID="52fe5e042c4c2aa8ad60ef9d51dac12745b96ff0c527b5d29f7e2f139aeed1c9" exitCode=0 Oct 30 17:52:27.643628 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.643592 4455 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 17:52:27.645276 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.645241 4455 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:52:27.645422 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.645388 4455 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:52:27.648374 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.648352 4455 generic.go:356] "Generic (PLEG): container finished" podID="cbda8c42-0f24-41a3-8a5f-727a4094d021" containerID="7ad9ed6de9de0aa9a5bc4eb6b9c192579c9616fce9ededfacdfc012a2f1cd4a2" exitCode=0 Oct 30 17:52:27.648374 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.648373 4455 generic.go:356] "Generic (PLEG): container finished" podID="cbda8c42-0f24-41a3-8a5f-727a4094d021" containerID="c4189f47f1700b01c969b9ed31403e2f3ebf028a83c5dd56709c34dc67f095fc" exitCode=0 Oct 30 17:52:27.648482 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.648378 4455 generic.go:356] "Generic (PLEG): container finished" podID="cbda8c42-0f24-41a3-8a5f-727a4094d021" containerID="7116774dce2cb7873e3327c79f608296cca6a2a6ef34a9c1d19d57753bb2091f" exitCode=0 Oct 30 17:52:27.648482 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.648384 4455 generic.go:356] "Generic (PLEG): container finished" podID="cbda8c42-0f24-41a3-8a5f-727a4094d021" containerID="d5396f27e8740f28710a9f671a35555e6aee1cb83b54c82fd8574d0fb3cd5554" exitCode=0 Oct 30 17:52:27.648482 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.648389 4455 generic.go:356] "Generic (PLEG): container finished" podID="cbda8c42-0f24-41a3-8a5f-727a4094d021" containerID="72a2e9c81a427f06c4f14bc4e3da6a7a517968ecc4100a3b9ad687ec60937fb9" exitCode=0 Oct 30 17:52:27.648482 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.648395 4455 generic.go:356] "Generic (PLEG): container finished" podID="cbda8c42-0f24-41a3-8a5f-727a4094d021" containerID="676de4088f0bdb1023774358ac66b2ad608fe36ff6c3645ed67ed420fbf91333" exitCode=0 Oct 30 17:52:27.657588 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.657540 4455 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:52:27.658135 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.658102 4455 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:52:27.669287 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.669268 4455 manager.go:324] Recovery completed Oct 30 17:52:27.683395 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.683374 4455 cpu_manager.go:226] "Starting" policy="none" Oct 30 17:52:27.683395 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.683390 4455 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 17:52:27.683538 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.683411 4455 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 17:52:27.683574 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.683536 4455 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 17:52:27.683574 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.683548 4455 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 17:52:27.683574 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.683563 4455 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 17:52:27.683574 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.683569 4455 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 17:52:27.683574 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.683574 4455 policy_none.go:50] "Start" Oct 30 17:52:27.683697 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.683583 4455 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 17:52:27.683697 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.683591 4455 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 17:52:27.690744 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.690726 4455 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 17:52:27.690744 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.690743 4455 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 17:52:27.690744 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.690750 4455 policy_none.go:44] "Start" Oct 30 17:52:27.694684 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.694652 4455 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 17:52:27.697150 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.697098 4455 manager.go:341] "Starting Device Plugin manager" Oct 30 17:52:27.697821 ip-10-0-130-84 kubenswrapper[4455]: E1030 17:52:27.697802 4455 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 17:52:27.697889 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.697824 4455 server.go:87] "Starting device plugin registration server" Oct 30 17:52:27.698150 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.698138 4455 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 17:52:27.698194 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.698153 4455 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 17:52:27.698237 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.698223 4455 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 17:52:27.698359 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.698309 4455 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 17:52:27.698427 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.698369 4455 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 17:52:27.698427 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.698376 4455 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 17:52:27.698843 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.698733 4455 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T17:52:27.698358101Z","UUID":"fa4bfac2-10ff-46ea-b43c-68581eb4e661","Handler":null,"Name":"","Endpoint":""} Oct 30 17:52:27.701353 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.701312 4455 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 17:52:27.701437 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.701372 4455 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 17:52:27.701639 ip-10-0-130-84 kubenswrapper[4455]: E1030 17:52:27.701618 4455 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 17:52:27.726853 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.726810 4455 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-84.ec2.internal"] Oct 30 17:52:27.727282 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.727254 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-84.ec2.internal" event={"ID":"da45dfd37aee5c345465266021945ad3","Type":"ContainerStarted","Data":"5d85c45e7c121a33349d8096f9531a16efbb1c1fb886dbd0b6262e3b790d64f5"} Oct 30 17:52:27.727282 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.727288 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-84.ec2.internal" event={"ID":"da45dfd37aee5c345465266021945ad3","Type":"ContainerStarted","Data":"90dd88d0b6a648f009bdfc07e21ba45f5aafa9255dfd19e15012a0509f9d72a0"} Oct 30 17:52:27.727595 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.727309 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal" event={"ID":"5ed893d4ed91b2f423e686d98f2bd57e","Type":"ContainerStarted","Data":"c3a9ad863613fae5f119ec157dea356a423b29cb4571a1efc14bddc8dc0e02ff"} Oct 30 17:52:27.727595 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.727343 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal" event={"ID":"5ed893d4ed91b2f423e686d98f2bd57e","Type":"ContainerDied","Data":"52fe5e042c4c2aa8ad60ef9d51dac12745b96ff0c527b5d29f7e2f139aeed1c9"} Oct 30 17:52:27.727595 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.727363 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal" event={"ID":"5ed893d4ed91b2f423e686d98f2bd57e","Type":"ContainerStarted","Data":"b286dc59e270f5f38fd27ef90603fbbd38a397f5db1f21bfa70e2e9caa3fe25e"} Oct 30 17:52:27.727595 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.727399 4455 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal" Oct 30 17:52:27.727595 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.727513 4455 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-84.ec2.internal" Oct 30 17:52:27.765154 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.765125 4455 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:52:27.765281 ip-10-0-130-84 kubenswrapper[4455]: E1030 17:52:27.765186 4455 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-84.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-84.ec2.internal" Oct 30 17:52:27.765281 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.765125 4455 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:52:27.765281 ip-10-0-130-84 kubenswrapper[4455]: E1030 17:52:27.765253 4455 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal" Oct 30 17:52:27.798692 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.798655 4455 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:52:27.799982 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.799963 4455 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-84.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:52:27.799982 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.799996 4455 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-84.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:52:27.799982 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.800009 4455 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-84.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:52:27.800193 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.800046 4455 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-84.ec2.internal" Oct 30 17:52:27.832370 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.832348 4455 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-84.ec2.internal" Oct 30 17:52:27.832442 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.832416 4455 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-84.ec2.internal" Oct 30 17:52:27.845172 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.845109 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/da45dfd37aee5c345465266021945ad3-config\") pod \"kube-apiserver-proxy-ip-10-0-130-84.ec2.internal\" (UID: \"da45dfd37aee5c345465266021945ad3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-84.ec2.internal" Oct 30 17:52:27.845269 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.845184 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5ed893d4ed91b2f423e686d98f2bd57e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal\" (UID: \"5ed893d4ed91b2f423e686d98f2bd57e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal" Oct 30 17:52:27.845269 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.845206 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ed893d4ed91b2f423e686d98f2bd57e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal\" (UID: \"5ed893d4ed91b2f423e686d98f2bd57e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal" Oct 30 17:52:27.946166 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.946127 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/da45dfd37aee5c345465266021945ad3-config\") pod \"kube-apiserver-proxy-ip-10-0-130-84.ec2.internal\" (UID: \"da45dfd37aee5c345465266021945ad3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-84.ec2.internal" Oct 30 17:52:27.946166 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.946163 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5ed893d4ed91b2f423e686d98f2bd57e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal\" (UID: \"5ed893d4ed91b2f423e686d98f2bd57e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal" Oct 30 17:52:27.946364 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.946181 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ed893d4ed91b2f423e686d98f2bd57e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal\" (UID: \"5ed893d4ed91b2f423e686d98f2bd57e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal" Oct 30 17:52:27.946364 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.946232 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5ed893d4ed91b2f423e686d98f2bd57e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal\" (UID: \"5ed893d4ed91b2f423e686d98f2bd57e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal" Oct 30 17:52:27.946364 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.946243 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/da45dfd37aee5c345465266021945ad3-config\") pod \"kube-apiserver-proxy-ip-10-0-130-84.ec2.internal\" (UID: \"da45dfd37aee5c345465266021945ad3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-84.ec2.internal" Oct 30 17:52:27.946364 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:27.946278 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ed893d4ed91b2f423e686d98f2bd57e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal\" (UID: \"5ed893d4ed91b2f423e686d98f2bd57e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal" Oct 30 17:52:28.586599 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.586561 4455 apiserver.go:52] "Watching apiserver" Oct 30 17:52:28.663507 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.663479 4455 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal" Oct 30 17:52:28.672495 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.672464 4455 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 17:52:28.673597 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.673566 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-k75vq","openshift-multus/multus-sgp8n","kube-system/kube-apiserver-proxy-ip-10-0-130-84.ec2.internal","openshift-ingress-canary/ingress-canary-p96b8","openshift-dns/dns-default-jpjk4","openshift-cluster-node-tuning-operator/tuned-t7knk","openshift-insights/insights-runtime-extractor-jvc4q","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal","kube-system/konnectivity-agent-qzhll","openshift-dns/node-resolver-bs5gk","openshift-network-operator/iptables-alerter-6826k","kube-system/global-pull-secret-syncer-pxv85","openshift-ovn-kubernetes/ovnkube-node-b4dtp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht","openshift-multus/multus-additional-cni-plugins-4bkwl","openshift-multus/network-metrics-daemon-stxf2","openshift-network-diagnostics/network-check-target-jgtj4"] Oct 30 17:52:28.675793 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.675769 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stxf2" Oct 30 17:52:28.678657 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.678639 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:28.685714 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.685696 4455 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:52:28.685793 ip-10-0-130-84 kubenswrapper[4455]: E1030 17:52:28.685744 4455 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-84.ec2.internal" Oct 30 17:52:28.691685 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.691669 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fzft4\"" Oct 30 17:52:28.693273 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.693243 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 17:52:28.693486 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.693470 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 17:52:28.693554 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.693500 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 17:52:28.693657 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.693593 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 17:52:28.693657 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.693597 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-dqbz4\"" Oct 30 17:52:28.693657 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.693606 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 17:52:28.693657 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.693615 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 17:52:28.693657 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.693600 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 17:52:28.693657 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.693652 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 17:52:28.693857 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.693696 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 17:52:28.693857 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.693721 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 17:52:28.693857 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.693737 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 17:52:28.694534 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.694516 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 17:52:28.694654 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.694535 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 17:52:28.694654 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.694540 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 17:52:28.694654 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.694565 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-k56gt\"" Oct 30 17:52:28.694654 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.694541 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ddhr5\"" Oct 30 17:52:28.694654 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.694583 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hhs6k\"" Oct 30 17:52:28.694654 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.694587 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 17:52:28.694654 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.694598 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5kfdv\"" Oct 30 17:52:28.694654 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.694642 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 17:52:28.694907 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.694732 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nb8hn\"" Oct 30 17:52:28.694907 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.694786 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:52:28.694907 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.694797 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zlfnf\"" Oct 30 17:52:28.694907 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.694859 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 17:52:28.695152 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.695128 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 17:52:28.695374 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.695363 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 17:52:28.695427 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.695410 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 17:52:28.696264 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.696249 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 17:52:28.696335 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.696280 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lsd9l\"" Oct 30 17:52:28.696486 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.696473 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 17:52:28.696521 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.696495 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 17:52:28.696708 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.696697 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 17:52:28.696949 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.696938 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:52:28.696996 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.696977 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-twdt2\"" Oct 30 17:52:28.697056 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.697045 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 17:52:28.697197 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.697186 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 17:52:28.697967 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.697956 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 17:52:28.698213 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.698191 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-k5mms\"" Oct 30 17:52:28.698262 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.698246 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-x8qzn\"" Oct 30 17:52:28.698773 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.698760 4455 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 17:52:28.700132 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.700120 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 17:52:28.700607 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.700594 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 17:52:28.701110 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.701093 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6xmc6\"" Oct 30 17:52:28.701158 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.701108 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 17:52:28.701158 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.701115 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 17:52:28.701231 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.701170 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cxbhw\"" Oct 30 17:52:28.703062 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.703050 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 17:52:28.703405 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.703391 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 17:52:28.703476 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.703397 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 17:52:28.703753 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.703725 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 17:52:28.703829 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.703787 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 17:52:28.703829 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.703797 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 17:52:28.704249 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.704231 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 17:52:28.704439 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.704415 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-99rd9\"" Oct 30 17:52:28.750702 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.750675 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cbda8c42-0f24-41a3-8a5f-727a4094d021-system-cni-dir\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:52:28.750702 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.750707 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-run-k8s-cni-cncf-io\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.750892 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.750734 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1fb157ba-42c2-4efc-a76b-6863fc987fb6-original-pull-secret\") pod \"global-pull-secret-syncer-pxv85\" (UID: \"1fb157ba-42c2-4efc-a76b-6863fc987fb6\") " pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:52:28.750892 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.750762 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b38a6e02-c51f-455b-b5bd-25ca428cddb2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvc4q\" (UID: \"b38a6e02-c51f-455b-b5bd-25ca428cddb2\") " pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:28.750892 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.750786 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13521cae-dc8a-4670-9321-61ad6ab3fd4d-metrics-certs\") pod \"network-metrics-daemon-stxf2\" (UID: \"13521cae-dc8a-4670-9321-61ad6ab3fd4d\") " pod="openshift-multus/network-metrics-daemon-stxf2" Oct 30 17:52:28.750892 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.750807 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-multus-socket-dir-parent\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.750892 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.750821 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-registration-dir\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:52:28.750892 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.750870 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-run-netns\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.751100 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.750908 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b38a6e02-c51f-455b-b5bd-25ca428cddb2-data-volume\") pod \"insights-runtime-extractor-jvc4q\" (UID: \"b38a6e02-c51f-455b-b5bd-25ca428cddb2\") " pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:28.751100 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.750929 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-system-cni-dir\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.751100 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.750961 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q25cz\" (UniqueName: \"kubernetes.io/projected/63b2e2e0-805f-4197-81d5-6eafe3831ded-kube-api-access-q25cz\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.751100 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.750987 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b38a6e02-c51f-455b-b5bd-25ca428cddb2-data-volume\") pod \"insights-runtime-extractor-jvc4q\" (UID: \"b38a6e02-c51f-455b-b5bd-25ca428cddb2\") " pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:28.751100 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.750992 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75cac6ab-2f87-4813-80a1-cc6efb29d187-host\") pod \"node-ca-k75vq\" (UID: \"75cac6ab-2f87-4813-80a1-cc6efb29d187\") " pod="openshift-image-registry/node-ca-k75vq" Oct 30 17:52:28.751100 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.751024 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1397e4e-e615-4954-8486-af559b65687e-tmp-dir\") pod \"node-resolver-bs5gk\" (UID: \"e1397e4e-e615-4954-8486-af559b65687e\") " pod="openshift-dns/node-resolver-bs5gk" Oct 30 17:52:28.751100 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.751065 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b38a6e02-c51f-455b-b5bd-25ca428cddb2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvc4q\" (UID: \"b38a6e02-c51f-455b-b5bd-25ca428cddb2\") " pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:28.751100 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.751064 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-sysconfig\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.751100 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.751089 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1397e4e-e615-4954-8486-af559b65687e-tmp-dir\") pod \"node-resolver-bs5gk\" (UID: \"e1397e4e-e615-4954-8486-af559b65687e\") " pod="openshift-dns/node-resolver-bs5gk" Oct 30 17:52:28.751533 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.751106 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-lib-modules\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.751533 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.751134 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-kubelet\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.751533 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.751204 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1fb157ba-42c2-4efc-a76b-6863fc987fb6-original-pull-secret\") pod \"global-pull-secret-syncer-pxv85\" (UID: \"1fb157ba-42c2-4efc-a76b-6863fc987fb6\") " pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:52:28.751533 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.751198 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7lk6\" (UniqueName: \"kubernetes.io/projected/b38a6e02-c51f-455b-b5bd-25ca428cddb2-kube-api-access-n7lk6\") pod \"insights-runtime-extractor-jvc4q\" (UID: \"b38a6e02-c51f-455b-b5bd-25ca428cddb2\") " pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:28.751533 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.751333 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mppvt\" (UniqueName: \"kubernetes.io/projected/cb68da46-11b6-48f4-b302-3ad63d93f82f-kube-api-access-mppvt\") pod \"network-check-target-jgtj4\" (UID: \"cb68da46-11b6-48f4-b302-3ad63d93f82f\") " pod="openshift-network-diagnostics/network-check-target-jgtj4" Oct 30 17:52:28.751533 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.751354 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnxpn\" (UniqueName: \"kubernetes.io/projected/c9e23597-79e7-4a28-ab52-fc8b343226ab-kube-api-access-lnxpn\") pod \"dns-default-jpjk4\" (UID: \"c9e23597-79e7-4a28-ab52-fc8b343226ab\") " pod="openshift-dns/dns-default-jpjk4" Oct 30 17:52:28.751533 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.751371 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-socket-dir\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:52:28.751533 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.751386 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-run-ovn\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.751533 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.751427 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbjzl\" (UniqueName: \"kubernetes.io/projected/75cac6ab-2f87-4813-80a1-cc6efb29d187-kube-api-access-pbjzl\") pod \"node-ca-k75vq\" (UID: \"75cac6ab-2f87-4813-80a1-cc6efb29d187\") " pod="openshift-image-registry/node-ca-k75vq" Oct 30 17:52:28.751533 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.751447 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rhhq\" (UniqueName: \"kubernetes.io/projected/8f348173-6e5e-404a-9126-7aa648f7eb01-kube-api-access-5rhhq\") pod \"iptables-alerter-6826k\" (UID: \"8f348173-6e5e-404a-9126-7aa648f7eb01\") " pod="openshift-network-operator/iptables-alerter-6826k" Oct 30 17:52:28.751533 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.751489 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9e23597-79e7-4a28-ab52-fc8b343226ab-tmp-dir\") pod \"dns-default-jpjk4\" (UID: \"c9e23597-79e7-4a28-ab52-fc8b343226ab\") " pod="openshift-dns/dns-default-jpjk4" Oct 30 17:52:28.751957 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.751610 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-hostroot\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.751957 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.751624 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9e23597-79e7-4a28-ab52-fc8b343226ab-tmp-dir\") pod \"dns-default-jpjk4\" (UID: \"c9e23597-79e7-4a28-ab52-fc8b343226ab\") " pod="openshift-dns/dns-default-jpjk4" Oct 30 17:52:28.751957 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.751653 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e1397e4e-e615-4954-8486-af559b65687e-hosts-file\") pod \"node-resolver-bs5gk\" (UID: \"e1397e4e-e615-4954-8486-af559b65687e\") " pod="openshift-dns/node-resolver-bs5gk" Oct 30 17:52:28.751957 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.751688 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f983c48d-08c4-41e2-af03-7c36cf651faa-env-overrides\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.751957 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.751714 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljxjz\" (UniqueName: \"kubernetes.io/projected/f983c48d-08c4-41e2-af03-7c36cf651faa-kube-api-access-ljxjz\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.751957 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.751740 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f348173-6e5e-404a-9126-7aa648f7eb01-host-slash\") pod \"iptables-alerter-6826k\" (UID: \"8f348173-6e5e-404a-9126-7aa648f7eb01\") " pod="openshift-network-operator/iptables-alerter-6826k" Oct 30 17:52:28.751957 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.751755 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-etc-kubernetes\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.751957 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.751769 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-sysctl-d\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.751957 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.751878 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-sysctl-conf\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.751957 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.751903 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f983c48d-08c4-41e2-af03-7c36cf651faa-env-overrides\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.751957 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.751911 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-log-socket\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.751957 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.751943 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f983c48d-08c4-41e2-af03-7c36cf651faa-ovnkube-config\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.752335 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.751978 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cbda8c42-0f24-41a3-8a5f-727a4094d021-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:52:28.752335 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.751997 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-run-multus-certs\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.752335 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.752086 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/75cac6ab-2f87-4813-80a1-cc6efb29d187-serviceca\") pod \"node-ca-k75vq\" (UID: \"75cac6ab-2f87-4813-80a1-cc6efb29d187\") " pod="openshift-image-registry/node-ca-k75vq" Oct 30 17:52:28.752335 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.752124 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/73dbe36c-9ccb-47d6-b11e-afc35b87008e-konnectivity-ca\") pod \"konnectivity-agent-qzhll\" (UID: \"73dbe36c-9ccb-47d6-b11e-afc35b87008e\") " pod="kube-system/konnectivity-agent-qzhll" Oct 30 17:52:28.752335 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.752180 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f983c48d-08c4-41e2-af03-7c36cf651faa-ovnkube-config\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.752335 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.752206 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cbda8c42-0f24-41a3-8a5f-727a4094d021-cnibin\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:52:28.752335 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.752236 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/63b2e2e0-805f-4197-81d5-6eafe3831ded-cni-binary-copy\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.752335 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.752259 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-var-lib-cni-bin\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.752335 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.752271 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/73dbe36c-9ccb-47d6-b11e-afc35b87008e-konnectivity-ca\") pod \"konnectivity-agent-qzhll\" (UID: \"73dbe36c-9ccb-47d6-b11e-afc35b87008e\") " pod="kube-system/konnectivity-agent-qzhll" Oct 30 17:52:28.752335 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.752292 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-var-lib-kubelet\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.752335 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.752333 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cbda8c42-0f24-41a3-8a5f-727a4094d021-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:52:28.752778 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.752347 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-multus-conf-dir\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.752778 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.752384 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-run\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.752778 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.752426 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l864g\" (UniqueName: \"kubernetes.io/projected/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-kube-api-access-l864g\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:52:28.752778 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.752453 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-slash\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.752778 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.752474 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-run-openvswitch\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.752778 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.752502 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/63b2e2e0-805f-4197-81d5-6eafe3831ded-cni-binary-copy\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.752778 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.752524 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-cni-bin\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.752778 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.752541 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.752778 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.752559 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f983c48d-08c4-41e2-af03-7c36cf651faa-ovnkube-script-lib\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.752778 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.752580 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-var-lib-kubelet\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.752778 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.752610 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-etc-selinux\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:52:28.752778 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.752694 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mww6s\" (UniqueName: \"kubernetes.io/projected/60c3d534-02fc-4665-ada2-e56fba531b87-kube-api-access-mww6s\") pod \"ingress-canary-p96b8\" (UID: \"60c3d534-02fc-4665-ada2-e56fba531b87\") " pod="openshift-ingress-canary/ingress-canary-p96b8" Oct 30 17:52:28.752778 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.752722 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-var-lib-openvswitch\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.753390 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.752801 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/75cac6ab-2f87-4813-80a1-cc6efb29d187-serviceca\") pod \"node-ca-k75vq\" (UID: \"75cac6ab-2f87-4813-80a1-cc6efb29d187\") " pod="openshift-image-registry/node-ca-k75vq" Oct 30 17:52:28.753390 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.752818 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-node-log\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.753390 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.752843 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f983c48d-08c4-41e2-af03-7c36cf651faa-ovnkube-script-lib\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.753390 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.752861 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f983c48d-08c4-41e2-af03-7c36cf651faa-ovn-node-metrics-cert\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.753390 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.752895 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b38a6e02-c51f-455b-b5bd-25ca428cddb2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvc4q\" (UID: \"b38a6e02-c51f-455b-b5bd-25ca428cddb2\") " pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:28.753390 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.752972 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e23597-79e7-4a28-ab52-fc8b343226ab-config-volume\") pod \"dns-default-jpjk4\" (UID: \"c9e23597-79e7-4a28-ab52-fc8b343226ab\") " pod="openshift-dns/dns-default-jpjk4" Oct 30 17:52:28.753390 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.753002 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9e23597-79e7-4a28-ab52-fc8b343226ab-metrics-tls\") pod \"dns-default-jpjk4\" (UID: \"c9e23597-79e7-4a28-ab52-fc8b343226ab\") " pod="openshift-dns/dns-default-jpjk4" Oct 30 17:52:28.753390 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.753032 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-run-netns\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.753390 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.753059 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1fb157ba-42c2-4efc-a76b-6863fc987fb6-kubelet-config\") pod \"global-pull-secret-syncer-pxv85\" (UID: \"1fb157ba-42c2-4efc-a76b-6863fc987fb6\") " pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:52:28.753390 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.753086 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2d2s\" (UniqueName: \"kubernetes.io/projected/e1397e4e-e615-4954-8486-af559b65687e-kube-api-access-b2d2s\") pod \"node-resolver-bs5gk\" (UID: \"e1397e4e-e615-4954-8486-af559b65687e\") " pod="openshift-dns/node-resolver-bs5gk" Oct 30 17:52:28.753390 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.753110 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-run-systemd\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.753390 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.753153 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-run-ovn-kubernetes\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.753390 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.753166 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e23597-79e7-4a28-ab52-fc8b343226ab-config-volume\") pod \"dns-default-jpjk4\" (UID: \"c9e23597-79e7-4a28-ab52-fc8b343226ab\") " pod="openshift-dns/dns-default-jpjk4" Oct 30 17:52:28.753390 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.753212 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cbda8c42-0f24-41a3-8a5f-727a4094d021-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:52:28.753390 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.753223 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9e23597-79e7-4a28-ab52-fc8b343226ab-metrics-tls\") pod \"dns-default-jpjk4\" (UID: \"c9e23597-79e7-4a28-ab52-fc8b343226ab\") " pod="openshift-dns/dns-default-jpjk4" Oct 30 17:52:28.753390 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.753244 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f983c48d-08c4-41e2-af03-7c36cf651faa-ovn-node-metrics-cert\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.753390 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.753253 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccflp\" (UniqueName: \"kubernetes.io/projected/cbda8c42-0f24-41a3-8a5f-727a4094d021-kube-api-access-ccflp\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:52:28.753390 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.753366 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-cnibin\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.753390 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.753395 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cbda8c42-0f24-41a3-8a5f-727a4094d021-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:52:28.753991 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.753449 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1fb157ba-42c2-4efc-a76b-6863fc987fb6-dbus\") pod \"global-pull-secret-syncer-pxv85\" (UID: \"1fb157ba-42c2-4efc-a76b-6863fc987fb6\") " pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:52:28.753991 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.753484 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-sys-fs\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:52:28.753991 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.753501 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60c3d534-02fc-4665-ada2-e56fba531b87-cert\") pod \"ingress-canary-p96b8\" (UID: \"60c3d534-02fc-4665-ada2-e56fba531b87\") " pod="openshift-ingress-canary/ingress-canary-p96b8" Oct 30 17:52:28.753991 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.753527 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-etc-openvswitch\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.753991 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.753608 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vbfw\" (UniqueName: \"kubernetes.io/projected/13521cae-dc8a-4670-9321-61ad6ab3fd4d-kube-api-access-5vbfw\") pod \"network-metrics-daemon-stxf2\" (UID: \"13521cae-dc8a-4670-9321-61ad6ab3fd4d\") " pod="openshift-multus/network-metrics-daemon-stxf2" Oct 30 17:52:28.753991 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.753667 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-systemd\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.753991 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.753688 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c126d55e-cb37-4070-904d-47f76d781751-tmp\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.753991 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.753689 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60c3d534-02fc-4665-ada2-e56fba531b87-cert\") pod \"ingress-canary-p96b8\" (UID: \"60c3d534-02fc-4665-ada2-e56fba531b87\") " pod="openshift-ingress-canary/ingress-canary-p96b8" Oct 30 17:52:28.753991 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.753703 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbw76\" (UniqueName: \"kubernetes.io/projected/c126d55e-cb37-4070-904d-47f76d781751-kube-api-access-sbw76\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.753991 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.753726 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-modprobe-d\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.753991 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.753756 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-kubernetes\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.753991 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.753803 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c126d55e-cb37-4070-904d-47f76d781751-tmp\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.753991 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.753860 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-host\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.753991 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.753895 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:52:28.753991 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.753923 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/73dbe36c-9ccb-47d6-b11e-afc35b87008e-agent-certs\") pod \"konnectivity-agent-qzhll\" (UID: \"73dbe36c-9ccb-47d6-b11e-afc35b87008e\") " pod="kube-system/konnectivity-agent-qzhll" Oct 30 17:52:28.753991 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.753946 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cbda8c42-0f24-41a3-8a5f-727a4094d021-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:52:28.754627 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.754041 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-multus-cni-dir\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.754627 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.754072 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c126d55e-cb37-4070-904d-47f76d781751-etc-tuned\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.754627 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.754100 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-cni-netd\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.754627 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.754150 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b38a6e02-c51f-455b-b5bd-25ca428cddb2-crio-socket\") pod \"insights-runtime-extractor-jvc4q\" (UID: \"b38a6e02-c51f-455b-b5bd-25ca428cddb2\") " pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:28.754627 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.754153 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c126d55e-cb37-4070-904d-47f76d781751-etc-tuned\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.754627 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.754172 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cbda8c42-0f24-41a3-8a5f-727a4094d021-cni-binary-copy\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:52:28.754627 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.754184 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/73dbe36c-9ccb-47d6-b11e-afc35b87008e-agent-certs\") pod \"konnectivity-agent-qzhll\" (UID: \"73dbe36c-9ccb-47d6-b11e-afc35b87008e\") " pod="kube-system/konnectivity-agent-qzhll" Oct 30 17:52:28.754627 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.754240 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8f348173-6e5e-404a-9126-7aa648f7eb01-iptables-alerter-script\") pod \"iptables-alerter-6826k\" (UID: \"8f348173-6e5e-404a-9126-7aa648f7eb01\") " pod="openshift-network-operator/iptables-alerter-6826k" Oct 30 17:52:28.754627 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.754270 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-var-lib-cni-multus\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.754627 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.754365 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/63b2e2e0-805f-4197-81d5-6eafe3831ded-multus-daemon-config\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.754627 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.754383 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cbda8c42-0f24-41a3-8a5f-727a4094d021-cni-binary-copy\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:52:28.754627 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.754445 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-sys\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.754627 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.754474 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-device-dir\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:52:28.754627 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.754485 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8f348173-6e5e-404a-9126-7aa648f7eb01-iptables-alerter-script\") pod \"iptables-alerter-6826k\" (UID: \"8f348173-6e5e-404a-9126-7aa648f7eb01\") " pod="openshift-network-operator/iptables-alerter-6826k" Oct 30 17:52:28.754627 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.754527 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-systemd-units\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.754627 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.754529 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/63b2e2e0-805f-4197-81d5-6eafe3831ded-multus-daemon-config\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.771704 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.771682 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mppvt\" (UniqueName: \"kubernetes.io/projected/cb68da46-11b6-48f4-b302-3ad63d93f82f-kube-api-access-mppvt\") pod \"network-check-target-jgtj4\" (UID: \"cb68da46-11b6-48f4-b302-3ad63d93f82f\") " pod="openshift-network-diagnostics/network-check-target-jgtj4" Oct 30 17:52:28.776391 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.776370 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rhhq\" (UniqueName: \"kubernetes.io/projected/8f348173-6e5e-404a-9126-7aa648f7eb01-kube-api-access-5rhhq\") pod \"iptables-alerter-6826k\" (UID: \"8f348173-6e5e-404a-9126-7aa648f7eb01\") " pod="openshift-network-operator/iptables-alerter-6826k" Oct 30 17:52:28.776596 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.776578 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnxpn\" (UniqueName: \"kubernetes.io/projected/c9e23597-79e7-4a28-ab52-fc8b343226ab-kube-api-access-lnxpn\") pod \"dns-default-jpjk4\" (UID: \"c9e23597-79e7-4a28-ab52-fc8b343226ab\") " pod="openshift-dns/dns-default-jpjk4" Oct 30 17:52:28.776646 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.776629 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljxjz\" (UniqueName: \"kubernetes.io/projected/f983c48d-08c4-41e2-af03-7c36cf651faa-kube-api-access-ljxjz\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.776836 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.776820 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7lk6\" (UniqueName: \"kubernetes.io/projected/b38a6e02-c51f-455b-b5bd-25ca428cddb2-kube-api-access-n7lk6\") pod \"insights-runtime-extractor-jvc4q\" (UID: \"b38a6e02-c51f-455b-b5bd-25ca428cddb2\") " pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:28.776948 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.776927 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q25cz\" (UniqueName: \"kubernetes.io/projected/63b2e2e0-805f-4197-81d5-6eafe3831ded-kube-api-access-q25cz\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.776979 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.776934 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbjzl\" (UniqueName: \"kubernetes.io/projected/75cac6ab-2f87-4813-80a1-cc6efb29d187-kube-api-access-pbjzl\") pod \"node-ca-k75vq\" (UID: \"75cac6ab-2f87-4813-80a1-cc6efb29d187\") " pod="openshift-image-registry/node-ca-k75vq" Oct 30 17:52:28.777596 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.777581 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbw76\" (UniqueName: \"kubernetes.io/projected/c126d55e-cb37-4070-904d-47f76d781751-kube-api-access-sbw76\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.777805 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.777790 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vbfw\" (UniqueName: \"kubernetes.io/projected/13521cae-dc8a-4670-9321-61ad6ab3fd4d-kube-api-access-5vbfw\") pod \"network-metrics-daemon-stxf2\" (UID: \"13521cae-dc8a-4670-9321-61ad6ab3fd4d\") " pod="openshift-multus/network-metrics-daemon-stxf2" Oct 30 17:52:28.779599 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.779583 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2d2s\" (UniqueName: \"kubernetes.io/projected/e1397e4e-e615-4954-8486-af559b65687e-kube-api-access-b2d2s\") pod \"node-resolver-bs5gk\" (UID: \"e1397e4e-e615-4954-8486-af559b65687e\") " pod="openshift-dns/node-resolver-bs5gk" Oct 30 17:52:28.782946 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.782931 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mww6s\" (UniqueName: \"kubernetes.io/projected/60c3d534-02fc-4665-ada2-e56fba531b87-kube-api-access-mww6s\") pod \"ingress-canary-p96b8\" (UID: \"60c3d534-02fc-4665-ada2-e56fba531b87\") " pod="openshift-ingress-canary/ingress-canary-p96b8" Oct 30 17:52:28.783074 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.783057 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccflp\" (UniqueName: \"kubernetes.io/projected/cbda8c42-0f24-41a3-8a5f-727a4094d021-kube-api-access-ccflp\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:52:28.784400 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.784386 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l864g\" (UniqueName: \"kubernetes.io/projected/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-kube-api-access-l864g\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:52:28.855808 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.855726 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-modprobe-d\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.855808 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.855760 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-kubernetes\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.855808 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.855777 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-host\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.855808 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.855791 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:52:28.856055 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.855829 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cbda8c42-0f24-41a3-8a5f-727a4094d021-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:52:28.856055 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.855871 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-multus-cni-dir\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.856055 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.855882 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-host\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.856055 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.855881 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cbda8c42-0f24-41a3-8a5f-727a4094d021-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:52:28.856055 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.855909 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-cni-netd\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.856055 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.855873 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-kubernetes\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.856055 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.855917 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-modprobe-d\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.856055 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.855911 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:52:28.856055 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.855934 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b38a6e02-c51f-455b-b5bd-25ca428cddb2-crio-socket\") pod \"insights-runtime-extractor-jvc4q\" (UID: \"b38a6e02-c51f-455b-b5bd-25ca428cddb2\") " pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:28.856055 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.855958 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-cni-netd\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.856055 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.855960 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-var-lib-cni-multus\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.856055 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.855970 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-multus-cni-dir\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.856055 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.855989 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-sys\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.856055 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.855996 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-var-lib-cni-multus\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.856055 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856007 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-device-dir\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:52:28.856055 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856033 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-sys\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.856055 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856044 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-systemd-units\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.856055 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856052 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b38a6e02-c51f-455b-b5bd-25ca428cddb2-crio-socket\") pod \"insights-runtime-extractor-jvc4q\" (UID: \"b38a6e02-c51f-455b-b5bd-25ca428cddb2\") " pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:28.856055 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856062 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cbda8c42-0f24-41a3-8a5f-727a4094d021-system-cni-dir\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856068 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-device-dir\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856078 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-run-k8s-cni-cncf-io\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856097 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-systemd-units\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856110 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cbda8c42-0f24-41a3-8a5f-727a4094d021-system-cni-dir\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856113 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-run-k8s-cni-cncf-io\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856141 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13521cae-dc8a-4670-9321-61ad6ab3fd4d-metrics-certs\") pod \"network-metrics-daemon-stxf2\" (UID: \"13521cae-dc8a-4670-9321-61ad6ab3fd4d\") " pod="openshift-multus/network-metrics-daemon-stxf2" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856158 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-multus-socket-dir-parent\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856190 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-registration-dir\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856212 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-run-netns\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856237 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-system-cni-dir\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856255 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-multus-socket-dir-parent\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856263 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75cac6ab-2f87-4813-80a1-cc6efb29d187-host\") pod \"node-ca-k75vq\" (UID: \"75cac6ab-2f87-4813-80a1-cc6efb29d187\") " pod="openshift-image-registry/node-ca-k75vq" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856278 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-registration-dir\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856288 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-sysconfig\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856294 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-system-cni-dir\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856281 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-run-netns\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856313 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-lib-modules\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856351 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75cac6ab-2f87-4813-80a1-cc6efb29d187-host\") pod \"node-ca-k75vq\" (UID: \"75cac6ab-2f87-4813-80a1-cc6efb29d187\") " pod="openshift-image-registry/node-ca-k75vq" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856370 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-sysconfig\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856388 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-kubelet\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856416 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-socket-dir\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856421 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-kubelet\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856438 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-run-ovn\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856438 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-lib-modules\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856462 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-hostroot\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856475 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-socket-dir\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856486 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e1397e4e-e615-4954-8486-af559b65687e-hosts-file\") pod \"node-resolver-bs5gk\" (UID: \"e1397e4e-e615-4954-8486-af559b65687e\") " pod="openshift-dns/node-resolver-bs5gk" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856499 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-run-ovn\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856510 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f348173-6e5e-404a-9126-7aa648f7eb01-host-slash\") pod \"iptables-alerter-6826k\" (UID: \"8f348173-6e5e-404a-9126-7aa648f7eb01\") " pod="openshift-network-operator/iptables-alerter-6826k" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856526 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-hostroot\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856540 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f348173-6e5e-404a-9126-7aa648f7eb01-host-slash\") pod \"iptables-alerter-6826k\" (UID: \"8f348173-6e5e-404a-9126-7aa648f7eb01\") " pod="openshift-network-operator/iptables-alerter-6826k" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856542 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-etc-kubernetes\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856552 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e1397e4e-e615-4954-8486-af559b65687e-hosts-file\") pod \"node-resolver-bs5gk\" (UID: \"e1397e4e-e615-4954-8486-af559b65687e\") " pod="openshift-dns/node-resolver-bs5gk" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856547 4455 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856598 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-etc-kubernetes\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856563 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-sysctl-d\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856637 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-sysctl-conf\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856652 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-sysctl-d\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856663 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-log-socket\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.856677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856692 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-run-multus-certs\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856716 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cbda8c42-0f24-41a3-8a5f-727a4094d021-cnibin\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856721 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-log-socket\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856735 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-var-lib-cni-bin\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856758 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-var-lib-kubelet\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856765 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-sysctl-conf\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856767 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-run-multus-certs\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856796 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-multus-conf-dir\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856807 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-var-lib-cni-bin\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856810 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cbda8c42-0f24-41a3-8a5f-727a4094d021-cnibin\") pod \"multus-additional-cni-plugins-4bkwl\" (UID: \"cbda8c42-0f24-41a3-8a5f-727a4094d021\") " pod="openshift-multus/multus-additional-cni-plugins-4bkwl" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856824 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-run\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856840 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-var-lib-kubelet\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856849 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-multus-conf-dir\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856849 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-slash\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856874 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-slash\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856879 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-run-openvswitch\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856895 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-cni-bin\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856911 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856922 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-run-openvswitch\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856893 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-run\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856937 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-var-lib-kubelet\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856945 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-cni-bin\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856962 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-etc-selinux\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856965 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856983 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-var-lib-openvswitch\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.856997 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-node-log\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.857020 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-etc-selinux\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.857031 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-node-log\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.857031 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-var-lib-openvswitch\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.857053 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b38a6e02-c51f-455b-b5bd-25ca428cddb2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvc4q\" (UID: \"b38a6e02-c51f-455b-b5bd-25ca428cddb2\") " pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.857102 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-run-netns\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.857134 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1fb157ba-42c2-4efc-a76b-6863fc987fb6-kubelet-config\") pod \"global-pull-secret-syncer-pxv85\" (UID: \"1fb157ba-42c2-4efc-a76b-6863fc987fb6\") " pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.857140 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-host-run-netns\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.857153 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-run-systemd\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.857163 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-var-lib-kubelet\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.857190 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-run-systemd\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.857193 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1fb157ba-42c2-4efc-a76b-6863fc987fb6-kubelet-config\") pod \"global-pull-secret-syncer-pxv85\" (UID: \"1fb157ba-42c2-4efc-a76b-6863fc987fb6\") " pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.857219 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-run-ovn-kubernetes\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.857247 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-cnibin\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.857268 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-host-run-ovn-kubernetes\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.857271 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1fb157ba-42c2-4efc-a76b-6863fc987fb6-dbus\") pod \"global-pull-secret-syncer-pxv85\" (UID: \"1fb157ba-42c2-4efc-a76b-6863fc987fb6\") " pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.857296 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-sys-fs\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.857311 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-etc-openvswitch\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.857355 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-systemd\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.857317 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1fb157ba-42c2-4efc-a76b-6863fc987fb6-dbus\") pod \"global-pull-secret-syncer-pxv85\" (UID: \"1fb157ba-42c2-4efc-a76b-6863fc987fb6\") " pod="kube-system/global-pull-secret-syncer-pxv85" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.857378 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3b7db1b0-8abd-47fb-9df8-7caab8adfed6-sys-fs\") pod \"aws-ebs-csi-driver-node-pwnht\" (UID: \"3b7db1b0-8abd-47fb-9df8-7caab8adfed6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwnht" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.857377 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/63b2e2e0-805f-4197-81d5-6eafe3831ded-cnibin\") pod \"multus-sgp8n\" (UID: \"63b2e2e0-805f-4197-81d5-6eafe3831ded\") " pod="openshift-multus/multus-sgp8n" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.857408 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f983c48d-08c4-41e2-af03-7c36cf651faa-etc-openvswitch\") pod \"ovnkube-node-b4dtp\" (UID: \"f983c48d-08c4-41e2-af03-7c36cf651faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:28.858034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.857421 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c126d55e-cb37-4070-904d-47f76d781751-etc-systemd\") pod \"tuned-t7knk\" (UID: \"c126d55e-cb37-4070-904d-47f76d781751\") " pod="openshift-cluster-node-tuning-operator/tuned-t7knk" Oct 30 17:52:28.859307 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.858870 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13521cae-dc8a-4670-9321-61ad6ab3fd4d-metrics-certs\") pod \"network-metrics-daemon-stxf2\" (UID: \"13521cae-dc8a-4670-9321-61ad6ab3fd4d\") " pod="openshift-multus/network-metrics-daemon-stxf2" Oct 30 17:52:28.859402 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.859385 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b38a6e02-c51f-455b-b5bd-25ca428cddb2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvc4q\" (UID: \"b38a6e02-c51f-455b-b5bd-25ca428cddb2\") " pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:28.976451 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.976414 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stxf2" Oct 30 17:52:28.979809 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:28.979787 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jvc4q" Oct 30 17:52:29.292548 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:29.292519 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-stxf2"] Oct 30 17:52:29.296493 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:29.296456 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13521cae_dc8a_4670_9321_61ad6ab3fd4d.slice/crio-804228e0342246e9d3a2dc91ea6002d7dcfa00dedc71320e28849558df40097b WatchSource:0}: Error finding container 804228e0342246e9d3a2dc91ea6002d7dcfa00dedc71320e28849558df40097b: Status 404 returned error can't find the container with id 804228e0342246e9d3a2dc91ea6002d7dcfa00dedc71320e28849558df40097b Oct 30 17:52:29.298595 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:29.298579 4455 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:52:29.323799 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:29.323767 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jvc4q"] Oct 30 17:52:29.668514 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:29.668437 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jvc4q" event={"ID":"b38a6e02-c51f-455b-b5bd-25ca428cddb2","Type":"ContainerStarted","Data":"bfe43cdd0623179254491227f0e3cccb8df929bb2e102c49f7f20485957a8426"} Oct 30 17:52:29.668514 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:29.668492 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jvc4q" event={"ID":"b38a6e02-c51f-455b-b5bd-25ca428cddb2","Type":"ContainerStarted","Data":"00f923ab927866ea1954642ef22284a4a18d2c79d401e476ca465146e7fe3bc3"} Oct 30 17:52:29.674460 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:29.674416 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-stxf2" event={"ID":"13521cae-dc8a-4670-9321-61ad6ab3fd4d","Type":"ContainerStarted","Data":"804228e0342246e9d3a2dc91ea6002d7dcfa00dedc71320e28849558df40097b"} Oct 30 17:52:29.693985 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:29.693956 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jpjk4" Oct 30 17:52:30.679539 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:30.679448 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-stxf2" event={"ID":"13521cae-dc8a-4670-9321-61ad6ab3fd4d","Type":"ContainerStarted","Data":"041cbcb54c6b93ca6143b38925718c48078455e21f83d3fd8df8233405bf90c3"} Oct 30 17:52:30.679539 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:30.679512 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-stxf2" event={"ID":"13521cae-dc8a-4670-9321-61ad6ab3fd4d","Type":"ContainerStarted","Data":"43f352731b7db76d807ac9649f38c9ef021f74a283d9a0523fff12d565ffe777"} Oct 30 17:52:30.681304 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:30.681280 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jvc4q" event={"ID":"b38a6e02-c51f-455b-b5bd-25ca428cddb2","Type":"ContainerStarted","Data":"297427e2a73692cd7962251b387cf81f810fe5725d8a34a2adf29bfbb5e7acfe"} Oct 30 17:52:31.689220 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:31.689178 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jvc4q" event={"ID":"b38a6e02-c51f-455b-b5bd-25ca428cddb2","Type":"ContainerStarted","Data":"a793843e084d3efba78abc4caf64ccf9fa41db1512b81e12139c0a737befb989"} Oct 30 17:52:33.706264 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:33.706234 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jgtj4" Oct 30 17:52:34.022543 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:34.022517 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:36.577227 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:36.577192 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:36.577706 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:36.577367 4455 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:52:36.940872 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:36.940778 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b4dtp" Oct 30 17:52:38.174861 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.174833 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-nbj4c"] Oct 30 17:52:38.176903 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.176880 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-nbj4c" Oct 30 17:52:38.194290 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.194265 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 17:52:38.194625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.194591 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-285m7\"" Oct 30 17:52:38.205522 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.205499 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 17:52:38.206254 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.206236 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-nbj4c"] Oct 30 17:52:38.271112 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.271081 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dh7v6"] Oct 30 17:52:38.272998 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.272969 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c94ff96f7-5ln4b"] Oct 30 17:52:38.273129 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.273089 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dh7v6" Oct 30 17:52:38.278798 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.278775 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c94ff96f7-5ln4b" Oct 30 17:52:38.294644 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.294626 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 17:52:38.294891 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.294877 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-k2hqn\"" Oct 30 17:52:38.311523 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.311503 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dh7v6"] Oct 30 17:52:38.315197 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.315174 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/210dfba3-f3d3-48db-983d-112569110049-tmp\") pod \"downloads-6dd789c8b7-nbj4c\" (UID: \"210dfba3-f3d3-48db-983d-112569110049\") " pod="openshift-console/downloads-6dd789c8b7-nbj4c" Oct 30 17:52:38.315294 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.315213 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8l8k\" (UniqueName: \"kubernetes.io/projected/210dfba3-f3d3-48db-983d-112569110049-kube-api-access-c8l8k\") pod \"downloads-6dd789c8b7-nbj4c\" (UID: \"210dfba3-f3d3-48db-983d-112569110049\") " pod="openshift-console/downloads-6dd789c8b7-nbj4c" Oct 30 17:52:38.315655 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.315635 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 17:52:38.317349 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.317310 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c94ff96f7-5ln4b"] Oct 30 17:52:38.394663 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.394626 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 17:52:38.394838 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.394766 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 17:52:38.394838 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.394806 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vwtrh\"" Oct 30 17:52:38.394921 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.394878 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 17:52:38.415764 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.415732 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/210dfba3-f3d3-48db-983d-112569110049-tmp\") pod \"downloads-6dd789c8b7-nbj4c\" (UID: \"210dfba3-f3d3-48db-983d-112569110049\") " pod="openshift-console/downloads-6dd789c8b7-nbj4c" Oct 30 17:52:38.415871 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.415777 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8l8k\" (UniqueName: \"kubernetes.io/projected/210dfba3-f3d3-48db-983d-112569110049-kube-api-access-c8l8k\") pod \"downloads-6dd789c8b7-nbj4c\" (UID: \"210dfba3-f3d3-48db-983d-112569110049\") " pod="openshift-console/downloads-6dd789c8b7-nbj4c" Oct 30 17:52:38.415918 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.415896 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/423e7d1a-291e-4960-849f-3307d4293b4e-installation-pull-secrets\") pod \"image-registry-7c94ff96f7-5ln4b\" (UID: \"423e7d1a-291e-4960-849f-3307d4293b4e\") " pod="openshift-image-registry/image-registry-7c94ff96f7-5ln4b" Oct 30 17:52:38.415953 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.415930 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql4gp\" (UniqueName: \"kubernetes.io/projected/423e7d1a-291e-4960-849f-3307d4293b4e-kube-api-access-ql4gp\") pod \"image-registry-7c94ff96f7-5ln4b\" (UID: \"423e7d1a-291e-4960-849f-3307d4293b4e\") " pod="openshift-image-registry/image-registry-7c94ff96f7-5ln4b" Oct 30 17:52:38.415997 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.415952 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/423e7d1a-291e-4960-849f-3307d4293b4e-registry-tls\") pod \"image-registry-7c94ff96f7-5ln4b\" (UID: \"423e7d1a-291e-4960-849f-3307d4293b4e\") " pod="openshift-image-registry/image-registry-7c94ff96f7-5ln4b" Oct 30 17:52:38.415997 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.415973 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/423e7d1a-291e-4960-849f-3307d4293b4e-bound-sa-token\") pod \"image-registry-7c94ff96f7-5ln4b\" (UID: \"423e7d1a-291e-4960-849f-3307d4293b4e\") " pod="openshift-image-registry/image-registry-7c94ff96f7-5ln4b" Oct 30 17:52:38.416070 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.416044 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/423e7d1a-291e-4960-849f-3307d4293b4e-trusted-ca\") pod \"image-registry-7c94ff96f7-5ln4b\" (UID: \"423e7d1a-291e-4960-849f-3307d4293b4e\") " pod="openshift-image-registry/image-registry-7c94ff96f7-5ln4b" Oct 30 17:52:38.416155 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.416137 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/210dfba3-f3d3-48db-983d-112569110049-tmp\") pod \"downloads-6dd789c8b7-nbj4c\" (UID: \"210dfba3-f3d3-48db-983d-112569110049\") " pod="openshift-console/downloads-6dd789c8b7-nbj4c" Oct 30 17:52:38.416193 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.416145 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/423e7d1a-291e-4960-849f-3307d4293b4e-image-registry-private-configuration\") pod \"image-registry-7c94ff96f7-5ln4b\" (UID: \"423e7d1a-291e-4960-849f-3307d4293b4e\") " pod="openshift-image-registry/image-registry-7c94ff96f7-5ln4b" Oct 30 17:52:38.416234 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.416198 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/423e7d1a-291e-4960-849f-3307d4293b4e-ca-trust-extracted\") pod \"image-registry-7c94ff96f7-5ln4b\" (UID: \"423e7d1a-291e-4960-849f-3307d4293b4e\") " pod="openshift-image-registry/image-registry-7c94ff96f7-5ln4b" Oct 30 17:52:38.416234 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.416218 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/423e7d1a-291e-4960-849f-3307d4293b4e-registry-certificates\") pod \"image-registry-7c94ff96f7-5ln4b\" (UID: \"423e7d1a-291e-4960-849f-3307d4293b4e\") " pod="openshift-image-registry/image-registry-7c94ff96f7-5ln4b" Oct 30 17:52:38.416300 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.416263 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/87ebbcab-1d74-4684-96d7-a61a85c39873-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-dh7v6\" (UID: \"87ebbcab-1d74-4684-96d7-a61a85c39873\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dh7v6" Oct 30 17:52:38.443891 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.443827 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8l8k\" (UniqueName: \"kubernetes.io/projected/210dfba3-f3d3-48db-983d-112569110049-kube-api-access-c8l8k\") pod \"downloads-6dd789c8b7-nbj4c\" (UID: \"210dfba3-f3d3-48db-983d-112569110049\") " pod="openshift-console/downloads-6dd789c8b7-nbj4c" Oct 30 17:52:38.487410 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.487377 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-nbj4c" Oct 30 17:52:38.516925 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.516892 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/423e7d1a-291e-4960-849f-3307d4293b4e-registry-tls\") pod \"image-registry-7c94ff96f7-5ln4b\" (UID: \"423e7d1a-291e-4960-849f-3307d4293b4e\") " pod="openshift-image-registry/image-registry-7c94ff96f7-5ln4b" Oct 30 17:52:38.517071 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.516938 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/423e7d1a-291e-4960-849f-3307d4293b4e-bound-sa-token\") pod \"image-registry-7c94ff96f7-5ln4b\" (UID: \"423e7d1a-291e-4960-849f-3307d4293b4e\") " pod="openshift-image-registry/image-registry-7c94ff96f7-5ln4b" Oct 30 17:52:38.517071 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.516969 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/423e7d1a-291e-4960-849f-3307d4293b4e-trusted-ca\") pod \"image-registry-7c94ff96f7-5ln4b\" (UID: \"423e7d1a-291e-4960-849f-3307d4293b4e\") " pod="openshift-image-registry/image-registry-7c94ff96f7-5ln4b" Oct 30 17:52:38.517071 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.517003 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/423e7d1a-291e-4960-849f-3307d4293b4e-image-registry-private-configuration\") pod \"image-registry-7c94ff96f7-5ln4b\" (UID: \"423e7d1a-291e-4960-849f-3307d4293b4e\") " pod="openshift-image-registry/image-registry-7c94ff96f7-5ln4b" Oct 30 17:52:38.517071 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.517039 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/423e7d1a-291e-4960-849f-3307d4293b4e-ca-trust-extracted\") pod \"image-registry-7c94ff96f7-5ln4b\" (UID: \"423e7d1a-291e-4960-849f-3307d4293b4e\") " pod="openshift-image-registry/image-registry-7c94ff96f7-5ln4b" Oct 30 17:52:38.517071 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.517064 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/423e7d1a-291e-4960-849f-3307d4293b4e-registry-certificates\") pod \"image-registry-7c94ff96f7-5ln4b\" (UID: \"423e7d1a-291e-4960-849f-3307d4293b4e\") " pod="openshift-image-registry/image-registry-7c94ff96f7-5ln4b" Oct 30 17:52:38.517354 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.517094 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/87ebbcab-1d74-4684-96d7-a61a85c39873-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-dh7v6\" (UID: \"87ebbcab-1d74-4684-96d7-a61a85c39873\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dh7v6" Oct 30 17:52:38.517354 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.517162 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/423e7d1a-291e-4960-849f-3307d4293b4e-installation-pull-secrets\") pod \"image-registry-7c94ff96f7-5ln4b\" (UID: \"423e7d1a-291e-4960-849f-3307d4293b4e\") " pod="openshift-image-registry/image-registry-7c94ff96f7-5ln4b" Oct 30 17:52:38.517354 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.517191 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql4gp\" (UniqueName: \"kubernetes.io/projected/423e7d1a-291e-4960-849f-3307d4293b4e-kube-api-access-ql4gp\") pod \"image-registry-7c94ff96f7-5ln4b\" (UID: \"423e7d1a-291e-4960-849f-3307d4293b4e\") " pod="openshift-image-registry/image-registry-7c94ff96f7-5ln4b" Oct 30 17:52:38.517574 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.517549 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/423e7d1a-291e-4960-849f-3307d4293b4e-ca-trust-extracted\") pod \"image-registry-7c94ff96f7-5ln4b\" (UID: \"423e7d1a-291e-4960-849f-3307d4293b4e\") " pod="openshift-image-registry/image-registry-7c94ff96f7-5ln4b" Oct 30 17:52:38.518383 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.518353 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/423e7d1a-291e-4960-849f-3307d4293b4e-registry-certificates\") pod \"image-registry-7c94ff96f7-5ln4b\" (UID: \"423e7d1a-291e-4960-849f-3307d4293b4e\") " pod="openshift-image-registry/image-registry-7c94ff96f7-5ln4b" Oct 30 17:52:38.518504 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.518490 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/423e7d1a-291e-4960-849f-3307d4293b4e-trusted-ca\") pod \"image-registry-7c94ff96f7-5ln4b\" (UID: \"423e7d1a-291e-4960-849f-3307d4293b4e\") " pod="openshift-image-registry/image-registry-7c94ff96f7-5ln4b" Oct 30 17:52:38.519735 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.519713 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/423e7d1a-291e-4960-849f-3307d4293b4e-registry-tls\") pod \"image-registry-7c94ff96f7-5ln4b\" (UID: \"423e7d1a-291e-4960-849f-3307d4293b4e\") " pod="openshift-image-registry/image-registry-7c94ff96f7-5ln4b" Oct 30 17:52:38.520301 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.520282 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/87ebbcab-1d74-4684-96d7-a61a85c39873-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-dh7v6\" (UID: \"87ebbcab-1d74-4684-96d7-a61a85c39873\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dh7v6" Oct 30 17:52:38.520301 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.520290 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/423e7d1a-291e-4960-849f-3307d4293b4e-installation-pull-secrets\") pod \"image-registry-7c94ff96f7-5ln4b\" (UID: \"423e7d1a-291e-4960-849f-3307d4293b4e\") " pod="openshift-image-registry/image-registry-7c94ff96f7-5ln4b" Oct 30 17:52:38.520457 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.520300 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/423e7d1a-291e-4960-849f-3307d4293b4e-image-registry-private-configuration\") pod \"image-registry-7c94ff96f7-5ln4b\" (UID: \"423e7d1a-291e-4960-849f-3307d4293b4e\") " pod="openshift-image-registry/image-registry-7c94ff96f7-5ln4b" Oct 30 17:52:38.543197 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.543119 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/423e7d1a-291e-4960-849f-3307d4293b4e-bound-sa-token\") pod \"image-registry-7c94ff96f7-5ln4b\" (UID: \"423e7d1a-291e-4960-849f-3307d4293b4e\") " pod="openshift-image-registry/image-registry-7c94ff96f7-5ln4b" Oct 30 17:52:38.546081 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.546047 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql4gp\" (UniqueName: \"kubernetes.io/projected/423e7d1a-291e-4960-849f-3307d4293b4e-kube-api-access-ql4gp\") pod \"image-registry-7c94ff96f7-5ln4b\" (UID: \"423e7d1a-291e-4960-849f-3307d4293b4e\") " pod="openshift-image-registry/image-registry-7c94ff96f7-5ln4b" Oct 30 17:52:38.585535 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.585504 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dh7v6" Oct 30 17:52:38.590314 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.590295 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c94ff96f7-5ln4b" Oct 30 17:52:38.696881 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.696814 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jpjk4" Oct 30 17:52:38.788287 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.788247 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-nbj4c"] Oct 30 17:52:38.792690 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:38.792665 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod210dfba3_f3d3_48db_983d_112569110049.slice/crio-e595c1c8df1d63963460af029e460698d44fdc42c8a84029ce0a61bc0ca3bdcb WatchSource:0}: Error finding container e595c1c8df1d63963460af029e460698d44fdc42c8a84029ce0a61bc0ca3bdcb: Status 404 returned error can't find the container with id e595c1c8df1d63963460af029e460698d44fdc42c8a84029ce0a61bc0ca3bdcb Oct 30 17:52:38.878195 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.878165 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dh7v6"] Oct 30 17:52:38.881643 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:38.881608 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ebbcab_1d74_4684_96d7_a61a85c39873.slice/crio-fd1f5f8c27b6dfca72aba384691653652c672af3b279586299f21e5ea0bbdc92 WatchSource:0}: Error finding container fd1f5f8c27b6dfca72aba384691653652c672af3b279586299f21e5ea0bbdc92: Status 404 returned error can't find the container with id fd1f5f8c27b6dfca72aba384691653652c672af3b279586299f21e5ea0bbdc92 Oct 30 17:52:38.882877 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:38.882852 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c94ff96f7-5ln4b"] Oct 30 17:52:38.886550 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:38.886524 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod423e7d1a_291e_4960_849f_3307d4293b4e.slice/crio-9702b886b4104c2f84c5477b3234e68612608c2b8a18b7c646ee958aaa823975 WatchSource:0}: Error finding container 9702b886b4104c2f84c5477b3234e68612608c2b8a18b7c646ee958aaa823975: Status 404 returned error can't find the container with id 9702b886b4104c2f84c5477b3234e68612608c2b8a18b7c646ee958aaa823975 Oct 30 17:52:39.245564 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.244385 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7df77857d7-shvsp"] Oct 30 17:52:39.248074 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.248051 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7df77857d7-shvsp" Oct 30 17:52:39.265177 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.265067 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 17:52:39.265177 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.265130 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 17:52:39.265177 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.265131 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 17:52:39.265431 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.265188 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 17:52:39.265431 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.265251 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 17:52:39.265431 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.265317 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7dg2w\"" Oct 30 17:52:39.276982 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.276935 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7df77857d7-shvsp"] Oct 30 17:52:39.323129 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.323091 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4651f7fa-6227-4a0b-a628-76bde7e15a8b-console-oauth-config\") pod \"console-7df77857d7-shvsp\" (UID: \"4651f7fa-6227-4a0b-a628-76bde7e15a8b\") " pod="openshift-console/console-7df77857d7-shvsp" Oct 30 17:52:39.323129 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.323138 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khq4x\" (UniqueName: \"kubernetes.io/projected/4651f7fa-6227-4a0b-a628-76bde7e15a8b-kube-api-access-khq4x\") pod \"console-7df77857d7-shvsp\" (UID: \"4651f7fa-6227-4a0b-a628-76bde7e15a8b\") " pod="openshift-console/console-7df77857d7-shvsp" Oct 30 17:52:39.323405 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.323184 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4651f7fa-6227-4a0b-a628-76bde7e15a8b-service-ca\") pod \"console-7df77857d7-shvsp\" (UID: \"4651f7fa-6227-4a0b-a628-76bde7e15a8b\") " pod="openshift-console/console-7df77857d7-shvsp" Oct 30 17:52:39.323405 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.323222 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4651f7fa-6227-4a0b-a628-76bde7e15a8b-tmp\") pod \"console-7df77857d7-shvsp\" (UID: \"4651f7fa-6227-4a0b-a628-76bde7e15a8b\") " pod="openshift-console/console-7df77857d7-shvsp" Oct 30 17:52:39.323405 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.323251 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4651f7fa-6227-4a0b-a628-76bde7e15a8b-oauth-serving-cert\") pod \"console-7df77857d7-shvsp\" (UID: \"4651f7fa-6227-4a0b-a628-76bde7e15a8b\") " pod="openshift-console/console-7df77857d7-shvsp" Oct 30 17:52:39.323405 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.323278 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4651f7fa-6227-4a0b-a628-76bde7e15a8b-console-serving-cert\") pod \"console-7df77857d7-shvsp\" (UID: \"4651f7fa-6227-4a0b-a628-76bde7e15a8b\") " pod="openshift-console/console-7df77857d7-shvsp" Oct 30 17:52:39.323405 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.323373 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4651f7fa-6227-4a0b-a628-76bde7e15a8b-console-config\") pod \"console-7df77857d7-shvsp\" (UID: \"4651f7fa-6227-4a0b-a628-76bde7e15a8b\") " pod="openshift-console/console-7df77857d7-shvsp" Oct 30 17:52:39.424433 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.424394 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4651f7fa-6227-4a0b-a628-76bde7e15a8b-console-config\") pod \"console-7df77857d7-shvsp\" (UID: \"4651f7fa-6227-4a0b-a628-76bde7e15a8b\") " pod="openshift-console/console-7df77857d7-shvsp" Oct 30 17:52:39.424615 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.424457 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4651f7fa-6227-4a0b-a628-76bde7e15a8b-console-oauth-config\") pod \"console-7df77857d7-shvsp\" (UID: \"4651f7fa-6227-4a0b-a628-76bde7e15a8b\") " pod="openshift-console/console-7df77857d7-shvsp" Oct 30 17:52:39.424615 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.424485 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khq4x\" (UniqueName: \"kubernetes.io/projected/4651f7fa-6227-4a0b-a628-76bde7e15a8b-kube-api-access-khq4x\") pod \"console-7df77857d7-shvsp\" (UID: \"4651f7fa-6227-4a0b-a628-76bde7e15a8b\") " pod="openshift-console/console-7df77857d7-shvsp" Oct 30 17:52:39.425250 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.425215 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4651f7fa-6227-4a0b-a628-76bde7e15a8b-console-config\") pod \"console-7df77857d7-shvsp\" (UID: \"4651f7fa-6227-4a0b-a628-76bde7e15a8b\") " pod="openshift-console/console-7df77857d7-shvsp" Oct 30 17:52:39.425461 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.425277 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4651f7fa-6227-4a0b-a628-76bde7e15a8b-service-ca\") pod \"console-7df77857d7-shvsp\" (UID: \"4651f7fa-6227-4a0b-a628-76bde7e15a8b\") " pod="openshift-console/console-7df77857d7-shvsp" Oct 30 17:52:39.425461 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.425360 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4651f7fa-6227-4a0b-a628-76bde7e15a8b-tmp\") pod \"console-7df77857d7-shvsp\" (UID: \"4651f7fa-6227-4a0b-a628-76bde7e15a8b\") " pod="openshift-console/console-7df77857d7-shvsp" Oct 30 17:52:39.425461 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.425391 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4651f7fa-6227-4a0b-a628-76bde7e15a8b-oauth-serving-cert\") pod \"console-7df77857d7-shvsp\" (UID: \"4651f7fa-6227-4a0b-a628-76bde7e15a8b\") " pod="openshift-console/console-7df77857d7-shvsp" Oct 30 17:52:39.425461 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.425425 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4651f7fa-6227-4a0b-a628-76bde7e15a8b-console-serving-cert\") pod \"console-7df77857d7-shvsp\" (UID: \"4651f7fa-6227-4a0b-a628-76bde7e15a8b\") " pod="openshift-console/console-7df77857d7-shvsp" Oct 30 17:52:39.425786 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.425692 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4651f7fa-6227-4a0b-a628-76bde7e15a8b-tmp\") pod \"console-7df77857d7-shvsp\" (UID: \"4651f7fa-6227-4a0b-a628-76bde7e15a8b\") " pod="openshift-console/console-7df77857d7-shvsp" Oct 30 17:52:39.426015 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.425978 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4651f7fa-6227-4a0b-a628-76bde7e15a8b-service-ca\") pod \"console-7df77857d7-shvsp\" (UID: \"4651f7fa-6227-4a0b-a628-76bde7e15a8b\") " pod="openshift-console/console-7df77857d7-shvsp" Oct 30 17:52:39.426259 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.426211 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4651f7fa-6227-4a0b-a628-76bde7e15a8b-oauth-serving-cert\") pod \"console-7df77857d7-shvsp\" (UID: \"4651f7fa-6227-4a0b-a628-76bde7e15a8b\") " pod="openshift-console/console-7df77857d7-shvsp" Oct 30 17:52:39.428119 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.428096 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4651f7fa-6227-4a0b-a628-76bde7e15a8b-console-oauth-config\") pod \"console-7df77857d7-shvsp\" (UID: \"4651f7fa-6227-4a0b-a628-76bde7e15a8b\") " pod="openshift-console/console-7df77857d7-shvsp" Oct 30 17:52:39.428338 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.428296 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4651f7fa-6227-4a0b-a628-76bde7e15a8b-console-serving-cert\") pod \"console-7df77857d7-shvsp\" (UID: \"4651f7fa-6227-4a0b-a628-76bde7e15a8b\") " pod="openshift-console/console-7df77857d7-shvsp" Oct 30 17:52:39.447404 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.447377 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khq4x\" (UniqueName: \"kubernetes.io/projected/4651f7fa-6227-4a0b-a628-76bde7e15a8b-kube-api-access-khq4x\") pod \"console-7df77857d7-shvsp\" (UID: \"4651f7fa-6227-4a0b-a628-76bde7e15a8b\") " pod="openshift-console/console-7df77857d7-shvsp" Oct 30 17:52:39.560945 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.560859 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7df77857d7-shvsp" Oct 30 17:52:39.714433 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.714384 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-nbj4c" event={"ID":"210dfba3-f3d3-48db-983d-112569110049","Type":"ContainerStarted","Data":"e595c1c8df1d63963460af029e460698d44fdc42c8a84029ce0a61bc0ca3bdcb"} Oct 30 17:52:39.716562 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.716532 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c94ff96f7-5ln4b" event={"ID":"423e7d1a-291e-4960-849f-3307d4293b4e","Type":"ContainerStarted","Data":"f114c649d2008b7c4e81b54630aa988f45ec39577880d6fc38b21159d7447189"} Oct 30 17:52:39.716684 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.716585 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c94ff96f7-5ln4b" event={"ID":"423e7d1a-291e-4960-849f-3307d4293b4e","Type":"ContainerStarted","Data":"9702b886b4104c2f84c5477b3234e68612608c2b8a18b7c646ee958aaa823975"} Oct 30 17:52:39.717309 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.717289 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c94ff96f7-5ln4b" Oct 30 17:52:39.718971 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.718933 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dh7v6" event={"ID":"87ebbcab-1d74-4684-96d7-a61a85c39873","Type":"ContainerStarted","Data":"fd1f5f8c27b6dfca72aba384691653652c672af3b279586299f21e5ea0bbdc92"} Oct 30 17:52:39.761136 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.761074 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7c94ff96f7-5ln4b" podStartSLOduration=1.761053773 podStartE2EDuration="1.761053773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:39.760259627 +0000 UTC m=+12.247896662" watchObservedRunningTime="2026-10-30 17:52:39.761053773 +0000 UTC m=+12.248690807" Oct 30 17:52:39.864921 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:39.864903 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7df77857d7-shvsp"] Oct 30 17:52:39.871451 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:39.871421 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4651f7fa_6227_4a0b_a628_76bde7e15a8b.slice/crio-5014a9f257f4ff3a6a44f131c9b03cea745cb066ada40cf901e3117e85a0f422 WatchSource:0}: Error finding container 5014a9f257f4ff3a6a44f131c9b03cea745cb066ada40cf901e3117e85a0f422: Status 404 returned error can't find the container with id 5014a9f257f4ff3a6a44f131c9b03cea745cb066ada40cf901e3117e85a0f422 Oct 30 17:52:40.727278 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:40.727236 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dh7v6" event={"ID":"87ebbcab-1d74-4684-96d7-a61a85c39873","Type":"ContainerStarted","Data":"864df093a78df2705eb761e829908c67a2500e013890f9e02736919f1c85335f"} Oct 30 17:52:40.727762 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:40.727648 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dh7v6" Oct 30 17:52:40.729707 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:40.729680 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7df77857d7-shvsp" event={"ID":"4651f7fa-6227-4a0b-a628-76bde7e15a8b","Type":"ContainerStarted","Data":"5014a9f257f4ff3a6a44f131c9b03cea745cb066ada40cf901e3117e85a0f422"} Oct 30 17:52:40.734534 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:40.734489 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dh7v6" Oct 30 17:52:40.767267 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:40.767225 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dh7v6" podStartSLOduration=1.790247806 podStartE2EDuration="2.767211773s" totalImagesPullingTime="976.963967ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:52:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:40.767114665 +0000 UTC m=+13.254751701" watchObservedRunningTime="2026-10-30 17:52:40.767211773 +0000 UTC m=+13.254848805" Oct 30 17:52:41.346010 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:41.345973 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-wxt9d"] Oct 30 17:52:41.349097 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:41.349066 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wxt9d" Oct 30 17:52:41.375291 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:41.374162 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 17:52:41.375291 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:41.374349 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 17:52:41.375291 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:41.374351 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 17:52:41.375291 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:41.374408 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 17:52:41.375291 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:41.374698 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 17:52:41.375291 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:41.374758 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-4vl5l\"" Oct 30 17:52:41.376034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:41.376010 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-wxt9d"] Oct 30 17:52:41.445215 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:41.445179 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33b2e16b-dd03-453f-94c4-0fb6cbed0701-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-wxt9d\" (UID: \"33b2e16b-dd03-453f-94c4-0fb6cbed0701\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wxt9d" Oct 30 17:52:41.445397 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:41.445229 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq9hw\" (UniqueName: \"kubernetes.io/projected/33b2e16b-dd03-453f-94c4-0fb6cbed0701-kube-api-access-bq9hw\") pod \"prometheus-operator-9cd89b74c-wxt9d\" (UID: \"33b2e16b-dd03-453f-94c4-0fb6cbed0701\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wxt9d" Oct 30 17:52:41.445397 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:41.445271 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33b2e16b-dd03-453f-94c4-0fb6cbed0701-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-wxt9d\" (UID: \"33b2e16b-dd03-453f-94c4-0fb6cbed0701\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wxt9d" Oct 30 17:52:41.445397 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:41.445317 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/33b2e16b-dd03-453f-94c4-0fb6cbed0701-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-wxt9d\" (UID: \"33b2e16b-dd03-453f-94c4-0fb6cbed0701\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wxt9d" Oct 30 17:52:41.546639 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:41.546605 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33b2e16b-dd03-453f-94c4-0fb6cbed0701-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-wxt9d\" (UID: \"33b2e16b-dd03-453f-94c4-0fb6cbed0701\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wxt9d" Oct 30 17:52:41.546779 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:41.546650 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq9hw\" (UniqueName: \"kubernetes.io/projected/33b2e16b-dd03-453f-94c4-0fb6cbed0701-kube-api-access-bq9hw\") pod \"prometheus-operator-9cd89b74c-wxt9d\" (UID: \"33b2e16b-dd03-453f-94c4-0fb6cbed0701\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wxt9d" Oct 30 17:52:41.546779 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:41.546678 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33b2e16b-dd03-453f-94c4-0fb6cbed0701-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-wxt9d\" (UID: \"33b2e16b-dd03-453f-94c4-0fb6cbed0701\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wxt9d" Oct 30 17:52:41.546779 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:41.546728 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/33b2e16b-dd03-453f-94c4-0fb6cbed0701-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-wxt9d\" (UID: \"33b2e16b-dd03-453f-94c4-0fb6cbed0701\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wxt9d" Oct 30 17:52:41.547945 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:41.547917 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33b2e16b-dd03-453f-94c4-0fb6cbed0701-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-wxt9d\" (UID: \"33b2e16b-dd03-453f-94c4-0fb6cbed0701\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wxt9d" Oct 30 17:52:41.549983 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:41.549957 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33b2e16b-dd03-453f-94c4-0fb6cbed0701-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-wxt9d\" (UID: \"33b2e16b-dd03-453f-94c4-0fb6cbed0701\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wxt9d" Oct 30 17:52:41.554859 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:41.554832 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/33b2e16b-dd03-453f-94c4-0fb6cbed0701-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-wxt9d\" (UID: \"33b2e16b-dd03-453f-94c4-0fb6cbed0701\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wxt9d" Oct 30 17:52:41.570286 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:41.570263 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq9hw\" (UniqueName: \"kubernetes.io/projected/33b2e16b-dd03-453f-94c4-0fb6cbed0701-kube-api-access-bq9hw\") pod \"prometheus-operator-9cd89b74c-wxt9d\" (UID: \"33b2e16b-dd03-453f-94c4-0fb6cbed0701\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wxt9d" Oct 30 17:52:41.663258 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:41.663177 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wxt9d" Oct 30 17:52:41.963575 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:41.963489 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-wxt9d"] Oct 30 17:52:41.967758 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:41.967725 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b2e16b_dd03_453f_94c4_0fb6cbed0701.slice/crio-4f83eab1c456a0bf374f6e2a54fc81aee51e0e460b2753b8c3c35d03f32c05e3 WatchSource:0}: Error finding container 4f83eab1c456a0bf374f6e2a54fc81aee51e0e460b2753b8c3c35d03f32c05e3: Status 404 returned error can't find the container with id 4f83eab1c456a0bf374f6e2a54fc81aee51e0e460b2753b8c3c35d03f32c05e3 Oct 30 17:52:42.743877 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:42.743833 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wxt9d" event={"ID":"33b2e16b-dd03-453f-94c4-0fb6cbed0701","Type":"ContainerStarted","Data":"4f83eab1c456a0bf374f6e2a54fc81aee51e0e460b2753b8c3c35d03f32c05e3"} Oct 30 17:52:43.750489 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:43.750448 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wxt9d" event={"ID":"33b2e16b-dd03-453f-94c4-0fb6cbed0701","Type":"ContainerStarted","Data":"870656dd4b9a08e6f8ac767f3a12ba81b28b95cba81e96a2a8fe8f0e0451ffcd"} Oct 30 17:52:43.750967 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:43.750513 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wxt9d" event={"ID":"33b2e16b-dd03-453f-94c4-0fb6cbed0701","Type":"ContainerStarted","Data":"b61c33c2da66df2c18e96411089086f5fd33e0679ef740e683366d9f3fa65b3b"} Oct 30 17:52:43.752179 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:43.752149 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7df77857d7-shvsp" event={"ID":"4651f7fa-6227-4a0b-a628-76bde7e15a8b","Type":"ContainerStarted","Data":"30028aa3f89baa56dbd75e198e0fdf3e87625ae8fd87a59cf9b8f25ba44e2e93"} Oct 30 17:52:43.791619 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:43.791566 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wxt9d" podStartSLOduration=1.514598945 podStartE2EDuration="2.791553542s" totalImagesPullingTime="1.276954597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:52:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:43.790146494 +0000 UTC m=+16.277783531" watchObservedRunningTime="2026-10-30 17:52:43.791553542 +0000 UTC m=+16.279190576" Oct 30 17:52:43.831217 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:43.831161 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7df77857d7-shvsp" podStartSLOduration=1.862685683 podStartE2EDuration="4.831145967s" totalImagesPullingTime="2.968460284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:52:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:43.830985409 +0000 UTC m=+16.318622444" watchObservedRunningTime="2026-10-30 17:52:43.831145967 +0000 UTC m=+16.318783000" Oct 30 17:52:45.965062 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:45.965028 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-hr4pf"] Oct 30 17:52:45.968592 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:45.968571 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hr4pf" Oct 30 17:52:45.997740 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:45.997711 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 17:52:45.997966 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:45.997948 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-fjhtd\"" Oct 30 17:52:45.998070 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:45.998002 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 17:52:45.998944 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:45.998916 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-hr4pf"] Oct 30 17:52:46.013477 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.013445 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5pgt"] Oct 30 17:52:46.015942 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.015915 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5pgt" Oct 30 17:52:46.025529 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.025509 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tflc2"] Oct 30 17:52:46.027732 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.027709 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tflc2" Oct 30 17:52:46.034873 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.034850 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 17:52:46.049189 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.049169 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 17:52:46.049349 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.049312 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 17:52:46.050187 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.050159 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5pgt"] Oct 30 17:52:46.050281 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.050267 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-2mkbx\"" Oct 30 17:52:46.083940 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.083910 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 17:52:46.084148 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.084126 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1aac1dd8-54d7-4df2-b435-e14842f19975-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-hr4pf\" (UID: \"1aac1dd8-54d7-4df2-b435-e14842f19975\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hr4pf" Oct 30 17:52:46.084198 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.084161 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1aac1dd8-54d7-4df2-b435-e14842f19975-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-hr4pf\" (UID: \"1aac1dd8-54d7-4df2-b435-e14842f19975\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hr4pf" Oct 30 17:52:46.084198 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.084182 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z69hh\" (UniqueName: \"kubernetes.io/projected/1aac1dd8-54d7-4df2-b435-e14842f19975-kube-api-access-z69hh\") pod \"openshift-state-metrics-7bd7489957-hr4pf\" (UID: \"1aac1dd8-54d7-4df2-b435-e14842f19975\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hr4pf" Oct 30 17:52:46.084368 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.084342 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1aac1dd8-54d7-4df2-b435-e14842f19975-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-hr4pf\" (UID: \"1aac1dd8-54d7-4df2-b435-e14842f19975\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hr4pf" Oct 30 17:52:46.089446 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.089417 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 17:52:46.089572 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.089545 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-58pv5\"" Oct 30 17:52:46.089670 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.089648 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 17:52:46.184718 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.184690 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1aac1dd8-54d7-4df2-b435-e14842f19975-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-hr4pf\" (UID: \"1aac1dd8-54d7-4df2-b435-e14842f19975\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hr4pf" Oct 30 17:52:46.184895 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.184725 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z69hh\" (UniqueName: \"kubernetes.io/projected/1aac1dd8-54d7-4df2-b435-e14842f19975-kube-api-access-z69hh\") pod \"openshift-state-metrics-7bd7489957-hr4pf\" (UID: \"1aac1dd8-54d7-4df2-b435-e14842f19975\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hr4pf" Oct 30 17:52:46.184895 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.184758 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2a2fc35-0948-4648-b98a-086d173ed454-metrics-client-ca\") pod \"node-exporter-tflc2\" (UID: \"d2a2fc35-0948-4648-b98a-086d173ed454\") " pod="openshift-monitoring/node-exporter-tflc2" Oct 30 17:52:46.184895 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.184784 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55zh7\" (UniqueName: \"kubernetes.io/projected/75d3a246-8a03-458d-b744-c7910ba39c07-kube-api-access-55zh7\") pod \"kube-state-metrics-5dc8d7d89b-m5pgt\" (UID: \"75d3a246-8a03-458d-b744-c7910ba39c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5pgt" Oct 30 17:52:46.184895 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.184806 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75d3a246-8a03-458d-b744-c7910ba39c07-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-m5pgt\" (UID: \"75d3a246-8a03-458d-b744-c7910ba39c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5pgt" Oct 30 17:52:46.184895 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.184827 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d2a2fc35-0948-4648-b98a-086d173ed454-node-exporter-wtmp\") pod \"node-exporter-tflc2\" (UID: \"d2a2fc35-0948-4648-b98a-086d173ed454\") " pod="openshift-monitoring/node-exporter-tflc2" Oct 30 17:52:46.184895 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.184847 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/75d3a246-8a03-458d-b744-c7910ba39c07-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-m5pgt\" (UID: \"75d3a246-8a03-458d-b744-c7910ba39c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5pgt" Oct 30 17:52:46.184895 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.184871 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9qcz\" (UniqueName: \"kubernetes.io/projected/d2a2fc35-0948-4648-b98a-086d173ed454-kube-api-access-q9qcz\") pod \"node-exporter-tflc2\" (UID: \"d2a2fc35-0948-4648-b98a-086d173ed454\") " pod="openshift-monitoring/node-exporter-tflc2" Oct 30 17:52:46.184895 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.184890 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d2a2fc35-0948-4648-b98a-086d173ed454-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tflc2\" (UID: \"d2a2fc35-0948-4648-b98a-086d173ed454\") " pod="openshift-monitoring/node-exporter-tflc2" Oct 30 17:52:46.185282 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.184907 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/75d3a246-8a03-458d-b744-c7910ba39c07-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-m5pgt\" (UID: \"75d3a246-8a03-458d-b744-c7910ba39c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5pgt" Oct 30 17:52:46.185282 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.184941 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1aac1dd8-54d7-4df2-b435-e14842f19975-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-hr4pf\" (UID: \"1aac1dd8-54d7-4df2-b435-e14842f19975\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hr4pf" Oct 30 17:52:46.185282 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.184966 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2a2fc35-0948-4648-b98a-086d173ed454-sys\") pod \"node-exporter-tflc2\" (UID: \"d2a2fc35-0948-4648-b98a-086d173ed454\") " pod="openshift-monitoring/node-exporter-tflc2" Oct 30 17:52:46.185282 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.184982 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d2a2fc35-0948-4648-b98a-086d173ed454-root\") pod \"node-exporter-tflc2\" (UID: \"d2a2fc35-0948-4648-b98a-086d173ed454\") " pod="openshift-monitoring/node-exporter-tflc2" Oct 30 17:52:46.185282 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.185002 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/75d3a246-8a03-458d-b744-c7910ba39c07-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-m5pgt\" (UID: \"75d3a246-8a03-458d-b744-c7910ba39c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5pgt" Oct 30 17:52:46.185282 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.185033 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/75d3a246-8a03-458d-b744-c7910ba39c07-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-m5pgt\" (UID: \"75d3a246-8a03-458d-b744-c7910ba39c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5pgt" Oct 30 17:52:46.185282 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.185049 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d2a2fc35-0948-4648-b98a-086d173ed454-node-exporter-tls\") pod \"node-exporter-tflc2\" (UID: \"d2a2fc35-0948-4648-b98a-086d173ed454\") " pod="openshift-monitoring/node-exporter-tflc2" Oct 30 17:52:46.185282 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.185073 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d2a2fc35-0948-4648-b98a-086d173ed454-node-exporter-textfile\") pod \"node-exporter-tflc2\" (UID: \"d2a2fc35-0948-4648-b98a-086d173ed454\") " pod="openshift-monitoring/node-exporter-tflc2" Oct 30 17:52:46.185282 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.185104 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1aac1dd8-54d7-4df2-b435-e14842f19975-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-hr4pf\" (UID: \"1aac1dd8-54d7-4df2-b435-e14842f19975\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hr4pf" Oct 30 17:52:46.185282 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.185207 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d2a2fc35-0948-4648-b98a-086d173ed454-node-exporter-accelerators-collector-config\") pod \"node-exporter-tflc2\" (UID: \"d2a2fc35-0948-4648-b98a-086d173ed454\") " pod="openshift-monitoring/node-exporter-tflc2" Oct 30 17:52:46.185723 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.185529 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1aac1dd8-54d7-4df2-b435-e14842f19975-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-hr4pf\" (UID: \"1aac1dd8-54d7-4df2-b435-e14842f19975\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hr4pf" Oct 30 17:52:46.187971 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.187943 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1aac1dd8-54d7-4df2-b435-e14842f19975-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-hr4pf\" (UID: \"1aac1dd8-54d7-4df2-b435-e14842f19975\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hr4pf" Oct 30 17:52:46.188278 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.188246 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1aac1dd8-54d7-4df2-b435-e14842f19975-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-hr4pf\" (UID: \"1aac1dd8-54d7-4df2-b435-e14842f19975\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hr4pf" Oct 30 17:52:46.208536 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.208505 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z69hh\" (UniqueName: \"kubernetes.io/projected/1aac1dd8-54d7-4df2-b435-e14842f19975-kube-api-access-z69hh\") pod \"openshift-state-metrics-7bd7489957-hr4pf\" (UID: \"1aac1dd8-54d7-4df2-b435-e14842f19975\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hr4pf" Oct 30 17:52:46.283719 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.283261 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hr4pf" Oct 30 17:52:46.286415 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.285533 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d2a2fc35-0948-4648-b98a-086d173ed454-node-exporter-wtmp\") pod \"node-exporter-tflc2\" (UID: \"d2a2fc35-0948-4648-b98a-086d173ed454\") " pod="openshift-monitoring/node-exporter-tflc2" Oct 30 17:52:46.286415 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.285584 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/75d3a246-8a03-458d-b744-c7910ba39c07-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-m5pgt\" (UID: \"75d3a246-8a03-458d-b744-c7910ba39c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5pgt" Oct 30 17:52:46.286415 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.285611 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9qcz\" (UniqueName: \"kubernetes.io/projected/d2a2fc35-0948-4648-b98a-086d173ed454-kube-api-access-q9qcz\") pod \"node-exporter-tflc2\" (UID: \"d2a2fc35-0948-4648-b98a-086d173ed454\") " pod="openshift-monitoring/node-exporter-tflc2" Oct 30 17:52:46.286415 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.285636 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d2a2fc35-0948-4648-b98a-086d173ed454-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tflc2\" (UID: \"d2a2fc35-0948-4648-b98a-086d173ed454\") " pod="openshift-monitoring/node-exporter-tflc2" Oct 30 17:52:46.286415 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.285664 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/75d3a246-8a03-458d-b744-c7910ba39c07-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-m5pgt\" (UID: \"75d3a246-8a03-458d-b744-c7910ba39c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5pgt" Oct 30 17:52:46.286415 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.285688 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d2a2fc35-0948-4648-b98a-086d173ed454-node-exporter-wtmp\") pod \"node-exporter-tflc2\" (UID: \"d2a2fc35-0948-4648-b98a-086d173ed454\") " pod="openshift-monitoring/node-exporter-tflc2" Oct 30 17:52:46.286415 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.285695 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2a2fc35-0948-4648-b98a-086d173ed454-sys\") pod \"node-exporter-tflc2\" (UID: \"d2a2fc35-0948-4648-b98a-086d173ed454\") " pod="openshift-monitoring/node-exporter-tflc2" Oct 30 17:52:46.286415 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.285749 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d2a2fc35-0948-4648-b98a-086d173ed454-root\") pod \"node-exporter-tflc2\" (UID: \"d2a2fc35-0948-4648-b98a-086d173ed454\") " pod="openshift-monitoring/node-exporter-tflc2" Oct 30 17:52:46.286415 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.285783 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/75d3a246-8a03-458d-b744-c7910ba39c07-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-m5pgt\" (UID: \"75d3a246-8a03-458d-b744-c7910ba39c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5pgt" Oct 30 17:52:46.286415 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.285823 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/75d3a246-8a03-458d-b744-c7910ba39c07-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-m5pgt\" (UID: \"75d3a246-8a03-458d-b744-c7910ba39c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5pgt" Oct 30 17:52:46.286415 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.285844 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d2a2fc35-0948-4648-b98a-086d173ed454-node-exporter-tls\") pod \"node-exporter-tflc2\" (UID: \"d2a2fc35-0948-4648-b98a-086d173ed454\") " pod="openshift-monitoring/node-exporter-tflc2" Oct 30 17:52:46.286415 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.285879 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d2a2fc35-0948-4648-b98a-086d173ed454-node-exporter-textfile\") pod \"node-exporter-tflc2\" (UID: \"d2a2fc35-0948-4648-b98a-086d173ed454\") " pod="openshift-monitoring/node-exporter-tflc2" Oct 30 17:52:46.286415 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.285910 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d2a2fc35-0948-4648-b98a-086d173ed454-node-exporter-accelerators-collector-config\") pod \"node-exporter-tflc2\" (UID: \"d2a2fc35-0948-4648-b98a-086d173ed454\") " pod="openshift-monitoring/node-exporter-tflc2" Oct 30 17:52:46.286415 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.285938 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2a2fc35-0948-4648-b98a-086d173ed454-metrics-client-ca\") pod \"node-exporter-tflc2\" (UID: \"d2a2fc35-0948-4648-b98a-086d173ed454\") " pod="openshift-monitoring/node-exporter-tflc2" Oct 30 17:52:46.286415 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.285962 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55zh7\" (UniqueName: \"kubernetes.io/projected/75d3a246-8a03-458d-b744-c7910ba39c07-kube-api-access-55zh7\") pod \"kube-state-metrics-5dc8d7d89b-m5pgt\" (UID: \"75d3a246-8a03-458d-b744-c7910ba39c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5pgt" Oct 30 17:52:46.286415 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.285991 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75d3a246-8a03-458d-b744-c7910ba39c07-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-m5pgt\" (UID: \"75d3a246-8a03-458d-b744-c7910ba39c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5pgt" Oct 30 17:52:46.286415 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.285998 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/75d3a246-8a03-458d-b744-c7910ba39c07-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-m5pgt\" (UID: \"75d3a246-8a03-458d-b744-c7910ba39c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5pgt" Oct 30 17:52:46.286415 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.286309 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2a2fc35-0948-4648-b98a-086d173ed454-sys\") pod \"node-exporter-tflc2\" (UID: \"d2a2fc35-0948-4648-b98a-086d173ed454\") " pod="openshift-monitoring/node-exporter-tflc2" Oct 30 17:52:46.287818 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.287142 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d2a2fc35-0948-4648-b98a-086d173ed454-node-exporter-accelerators-collector-config\") pod \"node-exporter-tflc2\" (UID: \"d2a2fc35-0948-4648-b98a-086d173ed454\") " pod="openshift-monitoring/node-exporter-tflc2" Oct 30 17:52:46.287818 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.287120 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/75d3a246-8a03-458d-b744-c7910ba39c07-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-m5pgt\" (UID: \"75d3a246-8a03-458d-b744-c7910ba39c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5pgt" Oct 30 17:52:46.287818 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.287469 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d2a2fc35-0948-4648-b98a-086d173ed454-node-exporter-textfile\") pod \"node-exporter-tflc2\" (UID: \"d2a2fc35-0948-4648-b98a-086d173ed454\") " pod="openshift-monitoring/node-exporter-tflc2" Oct 30 17:52:46.287818 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.287524 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d2a2fc35-0948-4648-b98a-086d173ed454-root\") pod \"node-exporter-tflc2\" (UID: \"d2a2fc35-0948-4648-b98a-086d173ed454\") " pod="openshift-monitoring/node-exporter-tflc2" Oct 30 17:52:46.287818 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.287797 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75d3a246-8a03-458d-b744-c7910ba39c07-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-m5pgt\" (UID: \"75d3a246-8a03-458d-b744-c7910ba39c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5pgt" Oct 30 17:52:46.288069 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.287817 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2a2fc35-0948-4648-b98a-086d173ed454-metrics-client-ca\") pod \"node-exporter-tflc2\" (UID: \"d2a2fc35-0948-4648-b98a-086d173ed454\") " pod="openshift-monitoring/node-exporter-tflc2" Oct 30 17:52:46.289309 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.289269 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/75d3a246-8a03-458d-b744-c7910ba39c07-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-m5pgt\" (UID: \"75d3a246-8a03-458d-b744-c7910ba39c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5pgt" Oct 30 17:52:46.289935 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.289912 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d2a2fc35-0948-4648-b98a-086d173ed454-node-exporter-tls\") pod \"node-exporter-tflc2\" (UID: \"d2a2fc35-0948-4648-b98a-086d173ed454\") " pod="openshift-monitoring/node-exporter-tflc2" Oct 30 17:52:46.290610 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.290587 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d2a2fc35-0948-4648-b98a-086d173ed454-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tflc2\" (UID: \"d2a2fc35-0948-4648-b98a-086d173ed454\") " pod="openshift-monitoring/node-exporter-tflc2" Oct 30 17:52:46.290733 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.290711 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/75d3a246-8a03-458d-b744-c7910ba39c07-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-m5pgt\" (UID: \"75d3a246-8a03-458d-b744-c7910ba39c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5pgt" Oct 30 17:52:46.308514 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.308453 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9qcz\" (UniqueName: \"kubernetes.io/projected/d2a2fc35-0948-4648-b98a-086d173ed454-kube-api-access-q9qcz\") pod \"node-exporter-tflc2\" (UID: \"d2a2fc35-0948-4648-b98a-086d173ed454\") " pod="openshift-monitoring/node-exporter-tflc2" Oct 30 17:52:46.309458 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.309406 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55zh7\" (UniqueName: \"kubernetes.io/projected/75d3a246-8a03-458d-b744-c7910ba39c07-kube-api-access-55zh7\") pod \"kube-state-metrics-5dc8d7d89b-m5pgt\" (UID: \"75d3a246-8a03-458d-b744-c7910ba39c07\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5pgt" Oct 30 17:52:46.332303 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.332265 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5pgt" Oct 30 17:52:46.343299 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.343267 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tflc2" Oct 30 17:52:46.365623 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:46.365134 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a2fc35_0948_4648_b98a_086d173ed454.slice/crio-7535275f385ecd5610f8db541c97604cc2bcdea776fe17d88b52beeb386d8768 WatchSource:0}: Error finding container 7535275f385ecd5610f8db541c97604cc2bcdea776fe17d88b52beeb386d8768: Status 404 returned error can't find the container with id 7535275f385ecd5610f8db541c97604cc2bcdea776fe17d88b52beeb386d8768 Oct 30 17:52:46.597867 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.597767 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-hr4pf"] Oct 30 17:52:46.601650 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:46.601622 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aac1dd8_54d7_4df2_b435_e14842f19975.slice/crio-6e23c1368c00ea527a56feb704a273a7544d9598524bc45fd96f6dddc142a10f WatchSource:0}: Error finding container 6e23c1368c00ea527a56feb704a273a7544d9598524bc45fd96f6dddc142a10f: Status 404 returned error can't find the container with id 6e23c1368c00ea527a56feb704a273a7544d9598524bc45fd96f6dddc142a10f Oct 30 17:52:46.663653 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.663629 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5pgt"] Oct 30 17:52:46.666457 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:46.666430 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d3a246_8a03_458d_b744_c7910ba39c07.slice/crio-aecaae1c1aeb755e0552fedb498796e1e6d82d38842ffb3bf4e0ac9f23ca94f6 WatchSource:0}: Error finding container aecaae1c1aeb755e0552fedb498796e1e6d82d38842ffb3bf4e0ac9f23ca94f6: Status 404 returned error can't find the container with id aecaae1c1aeb755e0552fedb498796e1e6d82d38842ffb3bf4e0ac9f23ca94f6 Oct 30 17:52:46.765000 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.764916 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tflc2" event={"ID":"d2a2fc35-0948-4648-b98a-086d173ed454","Type":"ContainerStarted","Data":"7535275f385ecd5610f8db541c97604cc2bcdea776fe17d88b52beeb386d8768"} Oct 30 17:52:46.767588 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.767507 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5pgt" event={"ID":"75d3a246-8a03-458d-b744-c7910ba39c07","Type":"ContainerStarted","Data":"aecaae1c1aeb755e0552fedb498796e1e6d82d38842ffb3bf4e0ac9f23ca94f6"} Oct 30 17:52:46.770362 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.770217 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hr4pf" event={"ID":"1aac1dd8-54d7-4df2-b435-e14842f19975","Type":"ContainerStarted","Data":"db289006bd1434364f7c0ce6140eea195f8c8d2bcb8afcc1a10c4981bfc7820a"} Oct 30 17:52:46.770362 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.770265 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hr4pf" event={"ID":"1aac1dd8-54d7-4df2-b435-e14842f19975","Type":"ContainerStarted","Data":"849210a1a8a13f53c90badad141eeda1479499bdb795eb31b422d412f61e0fa5"} Oct 30 17:52:46.770362 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:46.770292 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hr4pf" event={"ID":"1aac1dd8-54d7-4df2-b435-e14842f19975","Type":"ContainerStarted","Data":"6e23c1368c00ea527a56feb704a273a7544d9598524bc45fd96f6dddc142a10f"} Oct 30 17:52:47.381034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.381002 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 17:52:47.384573 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.384549 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.407833 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.407073 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 17:52:47.407833 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.407134 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 17:52:47.407833 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.407285 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 17:52:47.407833 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.407484 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-xdrrk\"" Oct 30 17:52:47.407833 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.407623 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 17:52:47.407833 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.407687 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 17:52:47.407833 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.407732 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 17:52:47.415580 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.415380 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 17:52:47.415580 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.415432 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 17:52:47.415580 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.415546 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 17:52:47.427489 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.427458 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 17:52:47.499280 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.499250 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-config-volume\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.499460 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.499288 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/efb931b6-9fc2-443a-ab5e-57e003cc56d3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.499460 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.499307 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.499460 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.499345 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9km54\" (UniqueName: \"kubernetes.io/projected/efb931b6-9fc2-443a-ab5e-57e003cc56d3-kube-api-access-9km54\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.499622 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.499511 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.499622 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.499548 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efb931b6-9fc2-443a-ab5e-57e003cc56d3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.499622 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.499578 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.499773 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.499651 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.499773 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.499716 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/efb931b6-9fc2-443a-ab5e-57e003cc56d3-config-out\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.499773 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.499750 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/efb931b6-9fc2-443a-ab5e-57e003cc56d3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.499919 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.499779 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-web-config\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.499919 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.499824 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efb931b6-9fc2-443a-ab5e-57e003cc56d3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.499919 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.499891 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.600619 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.600583 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.600796 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.600640 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-config-volume\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.600796 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.600667 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/efb931b6-9fc2-443a-ab5e-57e003cc56d3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.600796 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.600716 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.600796 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.600741 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9km54\" (UniqueName: \"kubernetes.io/projected/efb931b6-9fc2-443a-ab5e-57e003cc56d3-kube-api-access-9km54\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.600796 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.600793 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.601046 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.600817 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efb931b6-9fc2-443a-ab5e-57e003cc56d3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.601046 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.600842 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.601046 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.600873 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.601046 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.600919 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/efb931b6-9fc2-443a-ab5e-57e003cc56d3-config-out\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.601046 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.600956 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/efb931b6-9fc2-443a-ab5e-57e003cc56d3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.601046 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.600994 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-web-config\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.601046 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.601020 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efb931b6-9fc2-443a-ab5e-57e003cc56d3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.602208 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.601880 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/efb931b6-9fc2-443a-ab5e-57e003cc56d3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.602208 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.601953 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efb931b6-9fc2-443a-ab5e-57e003cc56d3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.603141 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.602931 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efb931b6-9fc2-443a-ab5e-57e003cc56d3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.607083 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.606920 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-web-config\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.607083 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.606958 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.607916 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.607852 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-config-volume\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.608714 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.608206 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/efb931b6-9fc2-443a-ab5e-57e003cc56d3-config-out\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.608714 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.608335 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.608714 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.608351 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.608714 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.608673 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.608957 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.608778 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/efb931b6-9fc2-443a-ab5e-57e003cc56d3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.609518 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.609469 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.644456 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.644344 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9km54\" (UniqueName: \"kubernetes.io/projected/efb931b6-9fc2-443a-ab5e-57e003cc56d3-kube-api-access-9km54\") pod \"alertmanager-main-0\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.709949 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.709586 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:52:47.789805 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.789767 4455 generic.go:356] "Generic (PLEG): container finished" podID="d2a2fc35-0948-4648-b98a-086d173ed454" containerID="570644710ad412e788f5595e2bca742673441011ffbef8b9a08d30573e63432b" exitCode=0 Oct 30 17:52:47.789955 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:47.789849 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tflc2" event={"ID":"d2a2fc35-0948-4648-b98a-086d173ed454","Type":"ContainerDied","Data":"570644710ad412e788f5595e2bca742673441011ffbef8b9a08d30573e63432b"} Oct 30 17:52:48.044566 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.044530 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 17:52:48.049782 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:48.049728 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb931b6_9fc2_443a_ab5e_57e003cc56d3.slice/crio-847038e5b0db03dd90f5b80801b53f77786863bca99065d027187199e7621d23 WatchSource:0}: Error finding container 847038e5b0db03dd90f5b80801b53f77786863bca99065d027187199e7621d23: Status 404 returned error can't find the container with id 847038e5b0db03dd90f5b80801b53f77786863bca99065d027187199e7621d23 Oct 30 17:52:48.131217 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.131134 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b"] Oct 30 17:52:48.134395 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.134369 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" Oct 30 17:52:48.149492 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.149269 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 17:52:48.149492 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.149389 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-24c2t3p9758gh\"" Oct 30 17:52:48.149492 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.149418 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 17:52:48.149492 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.149469 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 17:52:48.150141 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.150115 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 17:52:48.150270 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.150174 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 17:52:48.150414 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.150396 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-cdcjg\"" Oct 30 17:52:48.168558 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.168532 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b"] Oct 30 17:52:48.309028 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.308939 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/526f0988-8961-458a-b22b-8b0b983d1e0a-secret-grpc-tls\") pod \"thanos-querier-6b65dfd95c-n9j9b\" (UID: \"526f0988-8961-458a-b22b-8b0b983d1e0a\") " pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" Oct 30 17:52:48.309149 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.309103 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/526f0988-8961-458a-b22b-8b0b983d1e0a-metrics-client-ca\") pod \"thanos-querier-6b65dfd95c-n9j9b\" (UID: \"526f0988-8961-458a-b22b-8b0b983d1e0a\") " pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" Oct 30 17:52:48.309210 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.309146 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/526f0988-8961-458a-b22b-8b0b983d1e0a-secret-thanos-querier-tls\") pod \"thanos-querier-6b65dfd95c-n9j9b\" (UID: \"526f0988-8961-458a-b22b-8b0b983d1e0a\") " pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" Oct 30 17:52:48.309267 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.309235 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/526f0988-8961-458a-b22b-8b0b983d1e0a-tmp\") pod \"thanos-querier-6b65dfd95c-n9j9b\" (UID: \"526f0988-8961-458a-b22b-8b0b983d1e0a\") " pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" Oct 30 17:52:48.309344 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.309277 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/526f0988-8961-458a-b22b-8b0b983d1e0a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6b65dfd95c-n9j9b\" (UID: \"526f0988-8961-458a-b22b-8b0b983d1e0a\") " pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" Oct 30 17:52:48.309344 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.309304 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/526f0988-8961-458a-b22b-8b0b983d1e0a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6b65dfd95c-n9j9b\" (UID: \"526f0988-8961-458a-b22b-8b0b983d1e0a\") " pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" Oct 30 17:52:48.309454 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.309345 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/526f0988-8961-458a-b22b-8b0b983d1e0a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6b65dfd95c-n9j9b\" (UID: \"526f0988-8961-458a-b22b-8b0b983d1e0a\") " pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" Oct 30 17:52:48.309454 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.309392 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/526f0988-8961-458a-b22b-8b0b983d1e0a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6b65dfd95c-n9j9b\" (UID: \"526f0988-8961-458a-b22b-8b0b983d1e0a\") " pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" Oct 30 17:52:48.309454 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.309434 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vjzr\" (UniqueName: \"kubernetes.io/projected/526f0988-8961-458a-b22b-8b0b983d1e0a-kube-api-access-6vjzr\") pod \"thanos-querier-6b65dfd95c-n9j9b\" (UID: \"526f0988-8961-458a-b22b-8b0b983d1e0a\") " pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" Oct 30 17:52:48.410662 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.410575 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/526f0988-8961-458a-b22b-8b0b983d1e0a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6b65dfd95c-n9j9b\" (UID: \"526f0988-8961-458a-b22b-8b0b983d1e0a\") " pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" Oct 30 17:52:48.410662 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.410629 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/526f0988-8961-458a-b22b-8b0b983d1e0a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6b65dfd95c-n9j9b\" (UID: \"526f0988-8961-458a-b22b-8b0b983d1e0a\") " pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" Oct 30 17:52:48.410662 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.410657 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/526f0988-8961-458a-b22b-8b0b983d1e0a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6b65dfd95c-n9j9b\" (UID: \"526f0988-8961-458a-b22b-8b0b983d1e0a\") " pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" Oct 30 17:52:48.411244 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.410708 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/526f0988-8961-458a-b22b-8b0b983d1e0a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6b65dfd95c-n9j9b\" (UID: \"526f0988-8961-458a-b22b-8b0b983d1e0a\") " pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" Oct 30 17:52:48.411244 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.410757 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vjzr\" (UniqueName: \"kubernetes.io/projected/526f0988-8961-458a-b22b-8b0b983d1e0a-kube-api-access-6vjzr\") pod \"thanos-querier-6b65dfd95c-n9j9b\" (UID: \"526f0988-8961-458a-b22b-8b0b983d1e0a\") " pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" Oct 30 17:52:48.411244 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.410782 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/526f0988-8961-458a-b22b-8b0b983d1e0a-secret-grpc-tls\") pod \"thanos-querier-6b65dfd95c-n9j9b\" (UID: \"526f0988-8961-458a-b22b-8b0b983d1e0a\") " pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" Oct 30 17:52:48.411244 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.410823 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/526f0988-8961-458a-b22b-8b0b983d1e0a-metrics-client-ca\") pod \"thanos-querier-6b65dfd95c-n9j9b\" (UID: \"526f0988-8961-458a-b22b-8b0b983d1e0a\") " pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" Oct 30 17:52:48.411244 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.410854 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/526f0988-8961-458a-b22b-8b0b983d1e0a-secret-thanos-querier-tls\") pod \"thanos-querier-6b65dfd95c-n9j9b\" (UID: \"526f0988-8961-458a-b22b-8b0b983d1e0a\") " pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" Oct 30 17:52:48.411244 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.410922 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/526f0988-8961-458a-b22b-8b0b983d1e0a-tmp\") pod \"thanos-querier-6b65dfd95c-n9j9b\" (UID: \"526f0988-8961-458a-b22b-8b0b983d1e0a\") " pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" Oct 30 17:52:48.411633 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.411334 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/526f0988-8961-458a-b22b-8b0b983d1e0a-tmp\") pod \"thanos-querier-6b65dfd95c-n9j9b\" (UID: \"526f0988-8961-458a-b22b-8b0b983d1e0a\") " pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" Oct 30 17:52:48.412263 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.411971 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/526f0988-8961-458a-b22b-8b0b983d1e0a-metrics-client-ca\") pod \"thanos-querier-6b65dfd95c-n9j9b\" (UID: \"526f0988-8961-458a-b22b-8b0b983d1e0a\") " pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" Oct 30 17:52:48.414366 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.414281 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/526f0988-8961-458a-b22b-8b0b983d1e0a-secret-grpc-tls\") pod \"thanos-querier-6b65dfd95c-n9j9b\" (UID: \"526f0988-8961-458a-b22b-8b0b983d1e0a\") " pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" Oct 30 17:52:48.414587 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.414563 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/526f0988-8961-458a-b22b-8b0b983d1e0a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6b65dfd95c-n9j9b\" (UID: \"526f0988-8961-458a-b22b-8b0b983d1e0a\") " pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" Oct 30 17:52:48.422824 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.422796 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/526f0988-8961-458a-b22b-8b0b983d1e0a-secret-thanos-querier-tls\") pod \"thanos-querier-6b65dfd95c-n9j9b\" (UID: \"526f0988-8961-458a-b22b-8b0b983d1e0a\") " pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" Oct 30 17:52:48.424355 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.424295 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/526f0988-8961-458a-b22b-8b0b983d1e0a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6b65dfd95c-n9j9b\" (UID: \"526f0988-8961-458a-b22b-8b0b983d1e0a\") " pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" Oct 30 17:52:48.424699 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.424649 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/526f0988-8961-458a-b22b-8b0b983d1e0a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6b65dfd95c-n9j9b\" (UID: \"526f0988-8961-458a-b22b-8b0b983d1e0a\") " pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" Oct 30 17:52:48.426823 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.426799 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/526f0988-8961-458a-b22b-8b0b983d1e0a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6b65dfd95c-n9j9b\" (UID: \"526f0988-8961-458a-b22b-8b0b983d1e0a\") " pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" Oct 30 17:52:48.448760 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.448723 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vjzr\" (UniqueName: \"kubernetes.io/projected/526f0988-8961-458a-b22b-8b0b983d1e0a-kube-api-access-6vjzr\") pod \"thanos-querier-6b65dfd95c-n9j9b\" (UID: \"526f0988-8961-458a-b22b-8b0b983d1e0a\") " pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" Oct 30 17:52:48.454540 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.454285 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" Oct 30 17:52:48.790921 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.790887 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b"] Oct 30 17:52:48.805678 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:48.796301 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod526f0988_8961_458a_b22b_8b0b983d1e0a.slice/crio-17fd5759d429b19995c861bc99876822335d9d0d09658e62b7a709ae66a6dde4 WatchSource:0}: Error finding container 17fd5759d429b19995c861bc99876822335d9d0d09658e62b7a709ae66a6dde4: Status 404 returned error can't find the container with id 17fd5759d429b19995c861bc99876822335d9d0d09658e62b7a709ae66a6dde4 Oct 30 17:52:48.805678 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.801015 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tflc2" event={"ID":"d2a2fc35-0948-4648-b98a-086d173ed454","Type":"ContainerStarted","Data":"8e043620601151a0f37b4a5af57492c17d29c848ebd2cd0740df19e8e37ac63f"} Oct 30 17:52:48.805678 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.801085 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tflc2" event={"ID":"d2a2fc35-0948-4648-b98a-086d173ed454","Type":"ContainerStarted","Data":"9dec062d04cda627e6ab59c52d19ece0f8af180ae02ddc8f1cc6c821384f0f1a"} Oct 30 17:52:48.805678 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.803744 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5pgt" event={"ID":"75d3a246-8a03-458d-b744-c7910ba39c07","Type":"ContainerStarted","Data":"92e0c7ae96098d04f17fda47e59edc368d8083b5c7e01b3715c3e8e7e2633e18"} Oct 30 17:52:48.805678 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.803794 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5pgt" event={"ID":"75d3a246-8a03-458d-b744-c7910ba39c07","Type":"ContainerStarted","Data":"0bda0e6683a61a450a8b6a0f361b76dc9856c6d760f9e4ab69411c3acab845f0"} Oct 30 17:52:48.805678 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.803870 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5pgt" event={"ID":"75d3a246-8a03-458d-b744-c7910ba39c07","Type":"ContainerStarted","Data":"95bc8ea5c9bd1ff69c3e3547f52dac1a6a41a489c76a81f5bb29e5b0a3306ee5"} Oct 30 17:52:48.810050 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.810025 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hr4pf" event={"ID":"1aac1dd8-54d7-4df2-b435-e14842f19975","Type":"ContainerStarted","Data":"1bb8bf0e98810f58af47cf81f7398b9d01fbdeb9412d8bf68f1b8805ef61af90"} Oct 30 17:52:48.814341 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.814303 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efb931b6-9fc2-443a-ab5e-57e003cc56d3","Type":"ContainerStarted","Data":"847038e5b0db03dd90f5b80801b53f77786863bca99065d027187199e7621d23"} Oct 30 17:52:48.845286 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.845231 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tflc2" podStartSLOduration=2.763568846 podStartE2EDuration="3.84521629s" totalImagesPullingTime="991.995268ms" totalInitContainerRuntime="89.652176ms" isStatefulPod=true podCreationTimestamp="2026-10-30 17:52:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:48.84500869 +0000 UTC m=+21.332645715" watchObservedRunningTime="2026-10-30 17:52:48.84521629 +0000 UTC m=+21.332853326" Oct 30 17:52:48.892156 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.892088 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hr4pf" podStartSLOduration=2.817190373 podStartE2EDuration="3.892070712s" totalImagesPullingTime="1.074880339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:52:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:48.891029464 +0000 UTC m=+21.378666499" watchObservedRunningTime="2026-10-30 17:52:48.892070712 +0000 UTC m=+21.379707747" Oct 30 17:52:48.935012 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:48.934957 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5pgt" podStartSLOduration=2.452866702 podStartE2EDuration="3.934938786s" totalImagesPullingTime="1.482072084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:52:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:48.934220469 +0000 UTC m=+21.421857503" watchObservedRunningTime="2026-10-30 17:52:48.934938786 +0000 UTC m=+21.422575835" Oct 30 17:52:49.561966 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:49.561916 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7df77857d7-shvsp" Oct 30 17:52:49.562475 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:49.561991 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7df77857d7-shvsp" Oct 30 17:52:49.567924 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:49.567895 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7df77857d7-shvsp" Oct 30 17:52:49.825724 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:49.825627 4455 generic.go:356] "Generic (PLEG): container finished" podID="efb931b6-9fc2-443a-ab5e-57e003cc56d3" containerID="85c5870e529178dd1f9118cf962378547cd8309cce3ba28134ce1683e4deefd0" exitCode=0 Oct 30 17:52:49.825883 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:49.825766 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efb931b6-9fc2-443a-ab5e-57e003cc56d3","Type":"ContainerDied","Data":"85c5870e529178dd1f9118cf962378547cd8309cce3ba28134ce1683e4deefd0"} Oct 30 17:52:49.828097 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:49.828072 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" event={"ID":"526f0988-8961-458a-b22b-8b0b983d1e0a","Type":"ContainerStarted","Data":"17fd5759d429b19995c861bc99876822335d9d0d09658e62b7a709ae66a6dde4"} Oct 30 17:52:49.832746 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:49.832724 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7df77857d7-shvsp" Oct 30 17:52:50.657818 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.657382 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-c676d786-wh25r"] Oct 30 17:52:50.660912 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.660884 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c676d786-wh25r" Oct 30 17:52:50.712312 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.712259 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 17:52:50.713623 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.713595 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c676d786-wh25r"] Oct 30 17:52:50.805139 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.805109 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-jd7b4"] Oct 30 17:52:50.808504 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.808478 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-jd7b4" Oct 30 17:52:50.825794 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.825753 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 17:52:50.825913 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.825795 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-c2g76\"" Oct 30 17:52:50.834859 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.834829 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-jd7b4"] Oct 30 17:52:50.836313 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.836289 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6trj6\" (UniqueName: \"kubernetes.io/projected/900dbe0b-374f-4188-9b35-5b4e7282ff1d-kube-api-access-6trj6\") pod \"console-c676d786-wh25r\" (UID: \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\") " pod="openshift-console/console-c676d786-wh25r" Oct 30 17:52:50.836452 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.836344 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/900dbe0b-374f-4188-9b35-5b4e7282ff1d-tmp\") pod \"console-c676d786-wh25r\" (UID: \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\") " pod="openshift-console/console-c676d786-wh25r" Oct 30 17:52:50.836822 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.836790 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/900dbe0b-374f-4188-9b35-5b4e7282ff1d-console-oauth-config\") pod \"console-c676d786-wh25r\" (UID: \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\") " pod="openshift-console/console-c676d786-wh25r" Oct 30 17:52:50.836928 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.836862 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/900dbe0b-374f-4188-9b35-5b4e7282ff1d-console-config\") pod \"console-c676d786-wh25r\" (UID: \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\") " pod="openshift-console/console-c676d786-wh25r" Oct 30 17:52:50.836928 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.836899 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/900dbe0b-374f-4188-9b35-5b4e7282ff1d-trusted-ca-bundle\") pod \"console-c676d786-wh25r\" (UID: \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\") " pod="openshift-console/console-c676d786-wh25r" Oct 30 17:52:50.837005 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.836969 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/900dbe0b-374f-4188-9b35-5b4e7282ff1d-console-serving-cert\") pod \"console-c676d786-wh25r\" (UID: \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\") " pod="openshift-console/console-c676d786-wh25r" Oct 30 17:52:50.837037 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.837012 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/900dbe0b-374f-4188-9b35-5b4e7282ff1d-service-ca\") pod \"console-c676d786-wh25r\" (UID: \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\") " pod="openshift-console/console-c676d786-wh25r" Oct 30 17:52:50.837075 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.837066 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/900dbe0b-374f-4188-9b35-5b4e7282ff1d-oauth-serving-cert\") pod \"console-c676d786-wh25r\" (UID: \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\") " pod="openshift-console/console-c676d786-wh25r" Oct 30 17:52:50.941091 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.938125 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6trj6\" (UniqueName: \"kubernetes.io/projected/900dbe0b-374f-4188-9b35-5b4e7282ff1d-kube-api-access-6trj6\") pod \"console-c676d786-wh25r\" (UID: \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\") " pod="openshift-console/console-c676d786-wh25r" Oct 30 17:52:50.941091 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.938167 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/900dbe0b-374f-4188-9b35-5b4e7282ff1d-tmp\") pod \"console-c676d786-wh25r\" (UID: \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\") " pod="openshift-console/console-c676d786-wh25r" Oct 30 17:52:50.941091 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.938221 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/900dbe0b-374f-4188-9b35-5b4e7282ff1d-console-oauth-config\") pod \"console-c676d786-wh25r\" (UID: \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\") " pod="openshift-console/console-c676d786-wh25r" Oct 30 17:52:50.941091 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.938263 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/900dbe0b-374f-4188-9b35-5b4e7282ff1d-console-config\") pod \"console-c676d786-wh25r\" (UID: \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\") " pod="openshift-console/console-c676d786-wh25r" Oct 30 17:52:50.941091 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.938309 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/900dbe0b-374f-4188-9b35-5b4e7282ff1d-trusted-ca-bundle\") pod \"console-c676d786-wh25r\" (UID: \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\") " pod="openshift-console/console-c676d786-wh25r" Oct 30 17:52:50.941091 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.938355 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/15a3b1ff-f48e-41d8-b4cb-90cd252d940b-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-jd7b4\" (UID: \"15a3b1ff-f48e-41d8-b4cb-90cd252d940b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-jd7b4" Oct 30 17:52:50.941091 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.938396 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/900dbe0b-374f-4188-9b35-5b4e7282ff1d-console-serving-cert\") pod \"console-c676d786-wh25r\" (UID: \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\") " pod="openshift-console/console-c676d786-wh25r" Oct 30 17:52:50.941091 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.938421 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/900dbe0b-374f-4188-9b35-5b4e7282ff1d-service-ca\") pod \"console-c676d786-wh25r\" (UID: \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\") " pod="openshift-console/console-c676d786-wh25r" Oct 30 17:52:50.941091 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.938483 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/900dbe0b-374f-4188-9b35-5b4e7282ff1d-oauth-serving-cert\") pod \"console-c676d786-wh25r\" (UID: \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\") " pod="openshift-console/console-c676d786-wh25r" Oct 30 17:52:50.941091 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.938508 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnc7q\" (UniqueName: \"kubernetes.io/projected/15a3b1ff-f48e-41d8-b4cb-90cd252d940b-kube-api-access-lnc7q\") pod \"monitoring-plugin-77f4f76594-jd7b4\" (UID: \"15a3b1ff-f48e-41d8-b4cb-90cd252d940b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-jd7b4" Oct 30 17:52:50.941091 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.939554 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/900dbe0b-374f-4188-9b35-5b4e7282ff1d-tmp\") pod \"console-c676d786-wh25r\" (UID: \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\") " pod="openshift-console/console-c676d786-wh25r" Oct 30 17:52:50.941091 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.941033 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/900dbe0b-374f-4188-9b35-5b4e7282ff1d-console-config\") pod \"console-c676d786-wh25r\" (UID: \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\") " pod="openshift-console/console-c676d786-wh25r" Oct 30 17:52:50.941790 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.941162 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/900dbe0b-374f-4188-9b35-5b4e7282ff1d-trusted-ca-bundle\") pod \"console-c676d786-wh25r\" (UID: \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\") " pod="openshift-console/console-c676d786-wh25r" Oct 30 17:52:50.941790 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.941259 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/900dbe0b-374f-4188-9b35-5b4e7282ff1d-service-ca\") pod \"console-c676d786-wh25r\" (UID: \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\") " pod="openshift-console/console-c676d786-wh25r" Oct 30 17:52:50.941790 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.941288 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/900dbe0b-374f-4188-9b35-5b4e7282ff1d-oauth-serving-cert\") pod \"console-c676d786-wh25r\" (UID: \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\") " pod="openshift-console/console-c676d786-wh25r" Oct 30 17:52:50.944747 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.944700 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/900dbe0b-374f-4188-9b35-5b4e7282ff1d-console-serving-cert\") pod \"console-c676d786-wh25r\" (UID: \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\") " pod="openshift-console/console-c676d786-wh25r" Oct 30 17:52:50.946185 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.946139 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/900dbe0b-374f-4188-9b35-5b4e7282ff1d-console-oauth-config\") pod \"console-c676d786-wh25r\" (UID: \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\") " pod="openshift-console/console-c676d786-wh25r" Oct 30 17:52:50.967665 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.967621 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6trj6\" (UniqueName: \"kubernetes.io/projected/900dbe0b-374f-4188-9b35-5b4e7282ff1d-kube-api-access-6trj6\") pod \"console-c676d786-wh25r\" (UID: \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\") " pod="openshift-console/console-c676d786-wh25r" Oct 30 17:52:50.976008 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:50.975979 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c676d786-wh25r" Oct 30 17:52:51.040081 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:51.040049 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/15a3b1ff-f48e-41d8-b4cb-90cd252d940b-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-jd7b4\" (UID: \"15a3b1ff-f48e-41d8-b4cb-90cd252d940b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-jd7b4" Oct 30 17:52:51.041003 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:51.040652 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnc7q\" (UniqueName: \"kubernetes.io/projected/15a3b1ff-f48e-41d8-b4cb-90cd252d940b-kube-api-access-lnc7q\") pod \"monitoring-plugin-77f4f76594-jd7b4\" (UID: \"15a3b1ff-f48e-41d8-b4cb-90cd252d940b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-jd7b4" Oct 30 17:52:51.049666 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:51.049609 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/15a3b1ff-f48e-41d8-b4cb-90cd252d940b-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-jd7b4\" (UID: \"15a3b1ff-f48e-41d8-b4cb-90cd252d940b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-jd7b4" Oct 30 17:52:51.064744 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:51.064709 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnc7q\" (UniqueName: \"kubernetes.io/projected/15a3b1ff-f48e-41d8-b4cb-90cd252d940b-kube-api-access-lnc7q\") pod \"monitoring-plugin-77f4f76594-jd7b4\" (UID: \"15a3b1ff-f48e-41d8-b4cb-90cd252d940b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-jd7b4" Oct 30 17:52:51.136891 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:51.136639 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-jd7b4" Oct 30 17:52:51.301613 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:51.301571 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c676d786-wh25r"] Oct 30 17:52:51.309064 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:52:51.309027 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod900dbe0b_374f_4188_9b35_5b4e7282ff1d.slice/crio-811a7e2f05c9a6e35fa1c018c4eb1bf41fff45b4a69de52ee8b00d90024dd776 WatchSource:0}: Error finding container 811a7e2f05c9a6e35fa1c018c4eb1bf41fff45b4a69de52ee8b00d90024dd776: Status 404 returned error can't find the container with id 811a7e2f05c9a6e35fa1c018c4eb1bf41fff45b4a69de52ee8b00d90024dd776 Oct 30 17:52:51.453383 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:51.453283 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-jd7b4"] Oct 30 17:52:51.841971 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:51.841937 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efb931b6-9fc2-443a-ab5e-57e003cc56d3","Type":"ContainerStarted","Data":"0f8cb3eacd62f7f0094c805da392ec86d6ca3e32e3d2cc628c596eee1ccbab15"} Oct 30 17:52:51.842394 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:51.842056 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efb931b6-9fc2-443a-ab5e-57e003cc56d3","Type":"ContainerStarted","Data":"31e12d734fb8492037f5324ec732b0d744dfb30ab8cb32f403349e0c20348570"} Oct 30 17:52:51.845106 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:51.845083 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" event={"ID":"526f0988-8961-458a-b22b-8b0b983d1e0a","Type":"ContainerStarted","Data":"77ac7537420d7c3cf979dcacf42e72bd0e090fd2254c4ee7f02a1f1c880a9754"} Oct 30 17:52:51.845179 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:51.845141 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" event={"ID":"526f0988-8961-458a-b22b-8b0b983d1e0a","Type":"ContainerStarted","Data":"a11f07179cb6406bca74c66752bc3ee1e32d483f5c2d82e0306a5881c3c5a660"} Oct 30 17:52:51.845223 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:51.845182 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" event={"ID":"526f0988-8961-458a-b22b-8b0b983d1e0a","Type":"ContainerStarted","Data":"44af909c5b0993c52a63fdb1053d9b52952c29b3793267296295c7e7b06b7659"} Oct 30 17:52:51.849171 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:51.849146 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c676d786-wh25r" event={"ID":"900dbe0b-374f-4188-9b35-5b4e7282ff1d","Type":"ContainerStarted","Data":"054f0fd64a06441ce70dbcc616c2ae53824dbaf4c08562e1e43ab1eda50af52b"} Oct 30 17:52:51.849258 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:51.849193 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c676d786-wh25r" event={"ID":"900dbe0b-374f-4188-9b35-5b4e7282ff1d","Type":"ContainerStarted","Data":"811a7e2f05c9a6e35fa1c018c4eb1bf41fff45b4a69de52ee8b00d90024dd776"} Oct 30 17:52:51.851534 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:51.851508 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-jd7b4" event={"ID":"15a3b1ff-f48e-41d8-b4cb-90cd252d940b","Type":"ContainerStarted","Data":"a51a60da3c3deb80d96704555128caa5e01b2d1d295d8b947707b60c87355fb6"} Oct 30 17:52:51.902242 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:51.902188 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-c676d786-wh25r" podStartSLOduration=1.902167661 podStartE2EDuration="1.902167661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:52:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:51.900853019 +0000 UTC m=+24.388490054" watchObservedRunningTime="2026-10-30 17:52:51.902167661 +0000 UTC m=+24.389804694" Oct 30 17:52:51.969373 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:51.969346 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qzhll" Oct 30 17:52:52.859529 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:52.859487 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efb931b6-9fc2-443a-ab5e-57e003cc56d3","Type":"ContainerStarted","Data":"22a74892463a1b031960f163705346a1d23615ad435300c572b654a0692ac1df"} Oct 30 17:52:52.859965 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:52.859576 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efb931b6-9fc2-443a-ab5e-57e003cc56d3","Type":"ContainerStarted","Data":"a878b23685252e9b053dd3da352dd9720ec2b6d74f46c3c0b269c6bfe2983a9e"} Oct 30 17:52:52.859965 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:52.859680 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efb931b6-9fc2-443a-ab5e-57e003cc56d3","Type":"ContainerStarted","Data":"b5ddf634ff70725c2cf974adf92251dc198ec7abfdc2035f99af6c3eb3aacef3"} Oct 30 17:52:52.859965 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:52.859732 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efb931b6-9fc2-443a-ab5e-57e003cc56d3","Type":"ContainerStarted","Data":"bf5d65c13d922359caef794353cd693c50ba97b4fa42d0c5c3e0e028ccaef72d"} Oct 30 17:52:52.862978 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:52.862950 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" event={"ID":"526f0988-8961-458a-b22b-8b0b983d1e0a","Type":"ContainerStarted","Data":"85150aca28560d627de5b3b651af5c29e4df0e948b406a6108b4f3945db4d680"} Oct 30 17:52:52.863104 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:52.863018 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" event={"ID":"526f0988-8961-458a-b22b-8b0b983d1e0a","Type":"ContainerStarted","Data":"55239368b0b78e7020596a5e2bf53b7b633875e0913a067c2dcb80727ace9780"} Oct 30 17:52:52.863104 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:52.863049 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" event={"ID":"526f0988-8961-458a-b22b-8b0b983d1e0a","Type":"ContainerStarted","Data":"4cbd75d4875d83cd5ecedf945f54c77a228460ccaf1eca82c8a73f26c1d74604"} Oct 30 17:52:52.938679 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:52.937568 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.142001659 podStartE2EDuration="5.93754946s" totalImagesPullingTime="2.783394758s" totalInitContainerRuntime="12.153043ms" isStatefulPod=false podCreationTimestamp="2026-10-30 17:52:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:52.925947962 +0000 UTC m=+25.413584998" watchObservedRunningTime="2026-10-30 17:52:52.93754946 +0000 UTC m=+25.425186559" Oct 30 17:52:52.976056 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:52.976006 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" podStartSLOduration=2.239797129 podStartE2EDuration="4.975965247s" totalImagesPullingTime="2.736168118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:52:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:52.973249874 +0000 UTC m=+25.460886908" watchObservedRunningTime="2026-10-30 17:52:52.975965247 +0000 UTC m=+25.463602284" Oct 30 17:52:53.870087 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:53.870050 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-jd7b4" event={"ID":"15a3b1ff-f48e-41d8-b4cb-90cd252d940b","Type":"ContainerStarted","Data":"fee92b5179146b5227bd30887bb5ec48ebbcedf4c5b022cc689cb5a179b1172a"} Oct 30 17:52:53.870803 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:53.870785 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-jd7b4" Oct 30 17:52:53.870874 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:53.870820 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" Oct 30 17:52:53.876395 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:53.876371 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-jd7b4" Oct 30 17:52:53.914569 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:53.914506 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-jd7b4" podStartSLOduration=2.506119815 podStartE2EDuration="3.914487501s" totalImagesPullingTime="1.408367686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:52:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:53.912702996 +0000 UTC m=+26.400340030" watchObservedRunningTime="2026-10-30 17:52:53.914487501 +0000 UTC m=+26.402124536" Oct 30 17:52:57.890735 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:57.890691 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-nbj4c" event={"ID":"210dfba3-f3d3-48db-983d-112569110049","Type":"ContainerStarted","Data":"156c263cb2e088650bbe67e60111e14bba56098372c2936ee88c5836dc161dfe"} Oct 30 17:52:57.891265 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:57.890804 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-nbj4c" Oct 30 17:52:57.906899 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:57.906865 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-nbj4c" Oct 30 17:52:57.958987 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:57.958921 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-nbj4c" podStartSLOduration=1.540956591 podStartE2EDuration="19.958900962s" totalImagesPullingTime="18.417944371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:52:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:57.943119512 +0000 UTC m=+30.430756526" watchObservedRunningTime="2026-10-30 17:52:57.958900962 +0000 UTC m=+30.446537997" Oct 30 17:52:59.716861 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:59.716823 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c676d786-wh25r"] Oct 30 17:52:59.776844 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:59.776803 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-f56c74c7-9xv4s"] Oct 30 17:52:59.807703 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:59.807669 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f56c74c7-9xv4s" Oct 30 17:52:59.825100 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:59.825069 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-oauth-serving-cert\") pod \"console-f56c74c7-9xv4s\" (UID: \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\") " pod="openshift-console/console-f56c74c7-9xv4s" Oct 30 17:52:59.825228 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:59.825128 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-console-oauth-config\") pod \"console-f56c74c7-9xv4s\" (UID: \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\") " pod="openshift-console/console-f56c74c7-9xv4s" Oct 30 17:52:59.825228 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:59.825193 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8lgt\" (UniqueName: \"kubernetes.io/projected/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-kube-api-access-j8lgt\") pod \"console-f56c74c7-9xv4s\" (UID: \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\") " pod="openshift-console/console-f56c74c7-9xv4s" Oct 30 17:52:59.825355 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:59.825278 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-service-ca\") pod \"console-f56c74c7-9xv4s\" (UID: \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\") " pod="openshift-console/console-f56c74c7-9xv4s" Oct 30 17:52:59.825355 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:59.825297 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-tmp\") pod \"console-f56c74c7-9xv4s\" (UID: \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\") " pod="openshift-console/console-f56c74c7-9xv4s" Oct 30 17:52:59.825499 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:59.825354 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-trusted-ca-bundle\") pod \"console-f56c74c7-9xv4s\" (UID: \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\") " pod="openshift-console/console-f56c74c7-9xv4s" Oct 30 17:52:59.825499 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:59.825423 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-console-config\") pod \"console-f56c74c7-9xv4s\" (UID: \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\") " pod="openshift-console/console-f56c74c7-9xv4s" Oct 30 17:52:59.825499 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:59.825465 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-console-serving-cert\") pod \"console-f56c74c7-9xv4s\" (UID: \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\") " pod="openshift-console/console-f56c74c7-9xv4s" Oct 30 17:52:59.830735 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:59.830685 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f56c74c7-9xv4s"] Oct 30 17:52:59.881780 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:59.881748 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6b65dfd95c-n9j9b" Oct 30 17:52:59.926362 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:59.926305 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-trusted-ca-bundle\") pod \"console-f56c74c7-9xv4s\" (UID: \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\") " pod="openshift-console/console-f56c74c7-9xv4s" Oct 30 17:52:59.926538 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:59.926415 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-console-config\") pod \"console-f56c74c7-9xv4s\" (UID: \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\") " pod="openshift-console/console-f56c74c7-9xv4s" Oct 30 17:52:59.926538 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:59.926458 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-console-serving-cert\") pod \"console-f56c74c7-9xv4s\" (UID: \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\") " pod="openshift-console/console-f56c74c7-9xv4s" Oct 30 17:52:59.926538 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:59.926526 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-oauth-serving-cert\") pod \"console-f56c74c7-9xv4s\" (UID: \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\") " pod="openshift-console/console-f56c74c7-9xv4s" Oct 30 17:52:59.926696 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:59.926590 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-console-oauth-config\") pod \"console-f56c74c7-9xv4s\" (UID: \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\") " pod="openshift-console/console-f56c74c7-9xv4s" Oct 30 17:52:59.926696 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:59.926614 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8lgt\" (UniqueName: \"kubernetes.io/projected/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-kube-api-access-j8lgt\") pod \"console-f56c74c7-9xv4s\" (UID: \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\") " pod="openshift-console/console-f56c74c7-9xv4s" Oct 30 17:52:59.926804 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:59.926710 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-service-ca\") pod \"console-f56c74c7-9xv4s\" (UID: \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\") " pod="openshift-console/console-f56c74c7-9xv4s" Oct 30 17:52:59.926804 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:59.926739 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-tmp\") pod \"console-f56c74c7-9xv4s\" (UID: \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\") " pod="openshift-console/console-f56c74c7-9xv4s" Oct 30 17:52:59.927089 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:59.927063 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-tmp\") pod \"console-f56c74c7-9xv4s\" (UID: \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\") " pod="openshift-console/console-f56c74c7-9xv4s" Oct 30 17:52:59.927386 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:59.927368 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-trusted-ca-bundle\") pod \"console-f56c74c7-9xv4s\" (UID: \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\") " pod="openshift-console/console-f56c74c7-9xv4s" Oct 30 17:52:59.928599 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:59.928572 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-console-config\") pod \"console-f56c74c7-9xv4s\" (UID: \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\") " pod="openshift-console/console-f56c74c7-9xv4s" Oct 30 17:52:59.928715 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:59.928614 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-oauth-serving-cert\") pod \"console-f56c74c7-9xv4s\" (UID: \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\") " pod="openshift-console/console-f56c74c7-9xv4s" Oct 30 17:52:59.929237 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:59.929215 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-service-ca\") pod \"console-f56c74c7-9xv4s\" (UID: \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\") " pod="openshift-console/console-f56c74c7-9xv4s" Oct 30 17:52:59.930514 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:59.930487 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-console-oauth-config\") pod \"console-f56c74c7-9xv4s\" (UID: \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\") " pod="openshift-console/console-f56c74c7-9xv4s" Oct 30 17:52:59.935135 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:59.935100 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-console-serving-cert\") pod \"console-f56c74c7-9xv4s\" (UID: \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\") " pod="openshift-console/console-f56c74c7-9xv4s" Oct 30 17:52:59.957363 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:52:59.957251 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8lgt\" (UniqueName: \"kubernetes.io/projected/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-kube-api-access-j8lgt\") pod \"console-f56c74c7-9xv4s\" (UID: \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\") " pod="openshift-console/console-f56c74c7-9xv4s" Oct 30 17:53:00.139804 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:00.139709 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f56c74c7-9xv4s" Oct 30 17:53:00.398170 ip-10-0-130-84 kubenswrapper[4455]: E1030 17:53:00.398033 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 17:53:00.480594 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:00.480557 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f56c74c7-9xv4s"] Oct 30 17:53:00.487231 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:53:00.487165 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc1fd6e2_d215_4f2a_9bff_cb363aa8909f.slice/crio-4c33faac5dfc6fd3a8001a9eab2862623a4fc44f47eb65fa6997dbb897d2394b WatchSource:0}: Error finding container 4c33faac5dfc6fd3a8001a9eab2862623a4fc44f47eb65fa6997dbb897d2394b: Status 404 returned error can't find the container with id 4c33faac5dfc6fd3a8001a9eab2862623a4fc44f47eb65fa6997dbb897d2394b Oct 30 17:53:00.908536 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:00.908493 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f56c74c7-9xv4s" event={"ID":"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f","Type":"ContainerStarted","Data":"bd07108aaee5af81ae0daf2bf821ed5b8f7490eb19629992f1b85f72fc3f8192"} Oct 30 17:53:00.909040 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:00.908606 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f56c74c7-9xv4s" event={"ID":"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f","Type":"ContainerStarted","Data":"4c33faac5dfc6fd3a8001a9eab2862623a4fc44f47eb65fa6997dbb897d2394b"} Oct 30 17:53:00.951647 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:00.951594 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-f56c74c7-9xv4s" podStartSLOduration=1.9515806919999998 podStartE2EDuration="1.951580692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:53:00.950977748 +0000 UTC m=+33.438614783" watchObservedRunningTime="2026-10-30 17:53:00.951580692 +0000 UTC m=+33.439217728" Oct 30 17:53:00.977052 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:00.977012 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c676d786-wh25r" Oct 30 17:53:01.743022 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:01.742989 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7c94ff96f7-5ln4b" Oct 30 17:53:03.710104 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:03.710070 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jgtj4" Oct 30 17:53:10.140030 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:10.139951 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f56c74c7-9xv4s" Oct 30 17:53:10.140030 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:10.140030 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f56c74c7-9xv4s" Oct 30 17:53:10.144845 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:10.144824 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f56c74c7-9xv4s" Oct 30 17:53:10.955266 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:10.955238 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f56c74c7-9xv4s" Oct 30 17:53:11.028796 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:11.028769 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7df77857d7-shvsp"] Oct 30 17:53:24.741484 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:24.741423 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-c676d786-wh25r" podUID="900dbe0b-374f-4188-9b35-5b4e7282ff1d" containerName="console" containerID="cri-o://054f0fd64a06441ce70dbcc616c2ae53824dbaf4c08562e1e43ab1eda50af52b" gracePeriod=15 Oct 30 17:53:25.007102 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:25.007071 4455 generic.go:356] "Generic (PLEG): container finished" podID="900dbe0b-374f-4188-9b35-5b4e7282ff1d" containerID="054f0fd64a06441ce70dbcc616c2ae53824dbaf4c08562e1e43ab1eda50af52b" exitCode=0 Oct 30 17:53:25.007231 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:25.007150 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c676d786-wh25r" event={"ID":"900dbe0b-374f-4188-9b35-5b4e7282ff1d","Type":"ContainerDied","Data":"054f0fd64a06441ce70dbcc616c2ae53824dbaf4c08562e1e43ab1eda50af52b"} Oct 30 17:53:25.007272 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:25.007234 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c676d786-wh25r" event={"ID":"900dbe0b-374f-4188-9b35-5b4e7282ff1d","Type":"ContainerDied","Data":"811a7e2f05c9a6e35fa1c018c4eb1bf41fff45b4a69de52ee8b00d90024dd776"} Oct 30 17:53:25.007272 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:25.007249 4455 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="811a7e2f05c9a6e35fa1c018c4eb1bf41fff45b4a69de52ee8b00d90024dd776" Oct 30 17:53:25.022119 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:25.022100 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c676d786-wh25r" Oct 30 17:53:25.158455 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:25.158418 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/900dbe0b-374f-4188-9b35-5b4e7282ff1d-tmp\" (UniqueName: \"kubernetes.io/empty-dir/900dbe0b-374f-4188-9b35-5b4e7282ff1d-tmp\") pod \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\" (UID: \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\") " Oct 30 17:53:25.158634 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:25.158475 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/900dbe0b-374f-4188-9b35-5b4e7282ff1d-service-ca\" (UniqueName: \"kubernetes.io/configmap/900dbe0b-374f-4188-9b35-5b4e7282ff1d-service-ca\") pod \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\" (UID: \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\") " Oct 30 17:53:25.158634 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:25.158534 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/900dbe0b-374f-4188-9b35-5b4e7282ff1d-kube-api-access-6trj6\" (UniqueName: \"kubernetes.io/projected/900dbe0b-374f-4188-9b35-5b4e7282ff1d-kube-api-access-6trj6\") pod \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\" (UID: \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\") " Oct 30 17:53:25.158634 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:25.158560 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/900dbe0b-374f-4188-9b35-5b4e7282ff1d-console-config\" (UniqueName: \"kubernetes.io/configmap/900dbe0b-374f-4188-9b35-5b4e7282ff1d-console-config\") pod \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\" (UID: \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\") " Oct 30 17:53:25.158634 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:25.158590 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/900dbe0b-374f-4188-9b35-5b4e7282ff1d-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/900dbe0b-374f-4188-9b35-5b4e7282ff1d-console-oauth-config\") pod \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\" (UID: \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\") " Oct 30 17:53:25.158634 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:25.158628 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/900dbe0b-374f-4188-9b35-5b4e7282ff1d-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/900dbe0b-374f-4188-9b35-5b4e7282ff1d-oauth-serving-cert\") pod \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\" (UID: \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\") " Oct 30 17:53:25.158880 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:25.158679 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/900dbe0b-374f-4188-9b35-5b4e7282ff1d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/900dbe0b-374f-4188-9b35-5b4e7282ff1d-trusted-ca-bundle\") pod \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\" (UID: \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\") " Oct 30 17:53:25.158880 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:25.158710 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/900dbe0b-374f-4188-9b35-5b4e7282ff1d-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/900dbe0b-374f-4188-9b35-5b4e7282ff1d-console-serving-cert\") pod \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\" (UID: \"900dbe0b-374f-4188-9b35-5b4e7282ff1d\") " Oct 30 17:53:25.158880 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:25.158724 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/900dbe0b-374f-4188-9b35-5b4e7282ff1d-tmp" pod "900dbe0b-374f-4188-9b35-5b4e7282ff1d" (UID: "900dbe0b-374f-4188-9b35-5b4e7282ff1d"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:53:25.159032 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:25.158932 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/900dbe0b-374f-4188-9b35-5b4e7282ff1d-service-ca" pod "900dbe0b-374f-4188-9b35-5b4e7282ff1d" (UID: "900dbe0b-374f-4188-9b35-5b4e7282ff1d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:53:25.159090 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:25.159039 4455 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/900dbe0b-374f-4188-9b35-5b4e7282ff1d-tmp\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:53:25.159141 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:25.159119 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/900dbe0b-374f-4188-9b35-5b4e7282ff1d-console-config" pod "900dbe0b-374f-4188-9b35-5b4e7282ff1d" (UID: "900dbe0b-374f-4188-9b35-5b4e7282ff1d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:53:25.159198 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:25.159157 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/900dbe0b-374f-4188-9b35-5b4e7282ff1d-oauth-serving-cert" pod "900dbe0b-374f-4188-9b35-5b4e7282ff1d" (UID: "900dbe0b-374f-4188-9b35-5b4e7282ff1d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:53:25.159253 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:25.159235 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/900dbe0b-374f-4188-9b35-5b4e7282ff1d-trusted-ca-bundle" pod "900dbe0b-374f-4188-9b35-5b4e7282ff1d" (UID: "900dbe0b-374f-4188-9b35-5b4e7282ff1d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:53:25.160991 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:25.160966 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/900dbe0b-374f-4188-9b35-5b4e7282ff1d-console-oauth-config" pod "900dbe0b-374f-4188-9b35-5b4e7282ff1d" (UID: "900dbe0b-374f-4188-9b35-5b4e7282ff1d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:53:25.161472 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:25.161454 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/900dbe0b-374f-4188-9b35-5b4e7282ff1d-kube-api-access-6trj6" pod "900dbe0b-374f-4188-9b35-5b4e7282ff1d" (UID: "900dbe0b-374f-4188-9b35-5b4e7282ff1d"). InnerVolumeSpecName "kube-api-access-6trj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:53:25.161556 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:25.161504 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/900dbe0b-374f-4188-9b35-5b4e7282ff1d-console-serving-cert" pod "900dbe0b-374f-4188-9b35-5b4e7282ff1d" (UID: "900dbe0b-374f-4188-9b35-5b4e7282ff1d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:53:25.260299 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:25.260214 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6trj6\" (UniqueName: \"kubernetes.io/projected/900dbe0b-374f-4188-9b35-5b4e7282ff1d-kube-api-access-6trj6\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:53:25.260299 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:25.260245 4455 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/900dbe0b-374f-4188-9b35-5b4e7282ff1d-console-config\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:53:25.260299 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:25.260255 4455 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/900dbe0b-374f-4188-9b35-5b4e7282ff1d-console-oauth-config\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:53:25.260299 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:25.260266 4455 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/900dbe0b-374f-4188-9b35-5b4e7282ff1d-oauth-serving-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:53:25.260299 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:25.260276 4455 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/900dbe0b-374f-4188-9b35-5b4e7282ff1d-trusted-ca-bundle\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:53:25.260299 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:25.260284 4455 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/900dbe0b-374f-4188-9b35-5b4e7282ff1d-console-serving-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:53:25.260299 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:25.260292 4455 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/900dbe0b-374f-4188-9b35-5b4e7282ff1d-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:53:26.011385 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:26.011354 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c676d786-wh25r" Oct 30 17:53:26.058810 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:26.058780 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c676d786-wh25r"] Oct 30 17:53:26.070301 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:26.070273 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c676d786-wh25r"] Oct 30 17:53:27.631116 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:27.631063 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="900dbe0b-374f-4188-9b35-5b4e7282ff1d" path="/var/lib/kubelet/pods/900dbe0b-374f-4188-9b35-5b4e7282ff1d/volumes" Oct 30 17:53:30.383249 ip-10-0-130-84 kubenswrapper[4455]: E1030 17:53:30.383210 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 17:53:33.866640 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:33.866607 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-77dbb5d6f-5t9t6"] Oct 30 17:53:33.867040 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:33.866925 4455 cpu_manager.go:414] "containerMap: removing container" podUID="900dbe0b-374f-4188-9b35-5b4e7282ff1d" containerName="console" Oct 30 17:53:33.867040 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:33.866936 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="900dbe0b-374f-4188-9b35-5b4e7282ff1d" containerName="console" Oct 30 17:53:33.867040 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:33.867009 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="900dbe0b-374f-4188-9b35-5b4e7282ff1d" containerName="console" Oct 30 17:53:33.901301 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:33.901273 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77dbb5d6f-5t9t6" Oct 30 17:53:33.936998 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:33.936968 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e182135-7cf7-4362-8f51-95b7b3173965-service-ca\") pod \"console-77dbb5d6f-5t9t6\" (UID: \"4e182135-7cf7-4362-8f51-95b7b3173965\") " pod="openshift-console/console-77dbb5d6f-5t9t6" Oct 30 17:53:33.936998 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:33.937009 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e182135-7cf7-4362-8f51-95b7b3173965-console-oauth-config\") pod \"console-77dbb5d6f-5t9t6\" (UID: \"4e182135-7cf7-4362-8f51-95b7b3173965\") " pod="openshift-console/console-77dbb5d6f-5t9t6" Oct 30 17:53:33.937207 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:33.937103 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e182135-7cf7-4362-8f51-95b7b3173965-trusted-ca-bundle\") pod \"console-77dbb5d6f-5t9t6\" (UID: \"4e182135-7cf7-4362-8f51-95b7b3173965\") " pod="openshift-console/console-77dbb5d6f-5t9t6" Oct 30 17:53:33.937265 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:33.937210 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e182135-7cf7-4362-8f51-95b7b3173965-console-serving-cert\") pod \"console-77dbb5d6f-5t9t6\" (UID: \"4e182135-7cf7-4362-8f51-95b7b3173965\") " pod="openshift-console/console-77dbb5d6f-5t9t6" Oct 30 17:53:33.937343 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:33.937265 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e182135-7cf7-4362-8f51-95b7b3173965-console-config\") pod \"console-77dbb5d6f-5t9t6\" (UID: \"4e182135-7cf7-4362-8f51-95b7b3173965\") " pod="openshift-console/console-77dbb5d6f-5t9t6" Oct 30 17:53:33.937405 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:33.937369 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9hhx\" (UniqueName: \"kubernetes.io/projected/4e182135-7cf7-4362-8f51-95b7b3173965-kube-api-access-c9hhx\") pod \"console-77dbb5d6f-5t9t6\" (UID: \"4e182135-7cf7-4362-8f51-95b7b3173965\") " pod="openshift-console/console-77dbb5d6f-5t9t6" Oct 30 17:53:33.937454 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:33.937405 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e182135-7cf7-4362-8f51-95b7b3173965-oauth-serving-cert\") pod \"console-77dbb5d6f-5t9t6\" (UID: \"4e182135-7cf7-4362-8f51-95b7b3173965\") " pod="openshift-console/console-77dbb5d6f-5t9t6" Oct 30 17:53:33.937525 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:33.937507 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e182135-7cf7-4362-8f51-95b7b3173965-tmp\") pod \"console-77dbb5d6f-5t9t6\" (UID: \"4e182135-7cf7-4362-8f51-95b7b3173965\") " pod="openshift-console/console-77dbb5d6f-5t9t6" Oct 30 17:53:33.941621 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:33.941598 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77dbb5d6f-5t9t6"] Oct 30 17:53:34.038145 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:34.038116 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e182135-7cf7-4362-8f51-95b7b3173965-console-serving-cert\") pod \"console-77dbb5d6f-5t9t6\" (UID: \"4e182135-7cf7-4362-8f51-95b7b3173965\") " pod="openshift-console/console-77dbb5d6f-5t9t6" Oct 30 17:53:34.038145 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:34.038148 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e182135-7cf7-4362-8f51-95b7b3173965-console-config\") pod \"console-77dbb5d6f-5t9t6\" (UID: \"4e182135-7cf7-4362-8f51-95b7b3173965\") " pod="openshift-console/console-77dbb5d6f-5t9t6" Oct 30 17:53:34.038398 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:34.038178 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9hhx\" (UniqueName: \"kubernetes.io/projected/4e182135-7cf7-4362-8f51-95b7b3173965-kube-api-access-c9hhx\") pod \"console-77dbb5d6f-5t9t6\" (UID: \"4e182135-7cf7-4362-8f51-95b7b3173965\") " pod="openshift-console/console-77dbb5d6f-5t9t6" Oct 30 17:53:34.038398 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:34.038213 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e182135-7cf7-4362-8f51-95b7b3173965-oauth-serving-cert\") pod \"console-77dbb5d6f-5t9t6\" (UID: \"4e182135-7cf7-4362-8f51-95b7b3173965\") " pod="openshift-console/console-77dbb5d6f-5t9t6" Oct 30 17:53:34.038398 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:34.038255 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e182135-7cf7-4362-8f51-95b7b3173965-tmp\") pod \"console-77dbb5d6f-5t9t6\" (UID: \"4e182135-7cf7-4362-8f51-95b7b3173965\") " pod="openshift-console/console-77dbb5d6f-5t9t6" Oct 30 17:53:34.038398 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:34.038287 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e182135-7cf7-4362-8f51-95b7b3173965-service-ca\") pod \"console-77dbb5d6f-5t9t6\" (UID: \"4e182135-7cf7-4362-8f51-95b7b3173965\") " pod="openshift-console/console-77dbb5d6f-5t9t6" Oct 30 17:53:34.038398 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:34.038314 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e182135-7cf7-4362-8f51-95b7b3173965-console-oauth-config\") pod \"console-77dbb5d6f-5t9t6\" (UID: \"4e182135-7cf7-4362-8f51-95b7b3173965\") " pod="openshift-console/console-77dbb5d6f-5t9t6" Oct 30 17:53:34.038677 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:34.038426 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e182135-7cf7-4362-8f51-95b7b3173965-trusted-ca-bundle\") pod \"console-77dbb5d6f-5t9t6\" (UID: \"4e182135-7cf7-4362-8f51-95b7b3173965\") " pod="openshift-console/console-77dbb5d6f-5t9t6" Oct 30 17:53:34.038738 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:34.038681 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e182135-7cf7-4362-8f51-95b7b3173965-tmp\") pod \"console-77dbb5d6f-5t9t6\" (UID: \"4e182135-7cf7-4362-8f51-95b7b3173965\") " pod="openshift-console/console-77dbb5d6f-5t9t6" Oct 30 17:53:34.039187 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:34.039097 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e182135-7cf7-4362-8f51-95b7b3173965-console-config\") pod \"console-77dbb5d6f-5t9t6\" (UID: \"4e182135-7cf7-4362-8f51-95b7b3173965\") " pod="openshift-console/console-77dbb5d6f-5t9t6" Oct 30 17:53:34.039187 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:34.039140 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e182135-7cf7-4362-8f51-95b7b3173965-oauth-serving-cert\") pod \"console-77dbb5d6f-5t9t6\" (UID: \"4e182135-7cf7-4362-8f51-95b7b3173965\") " pod="openshift-console/console-77dbb5d6f-5t9t6" Oct 30 17:53:34.039478 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:34.039463 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e182135-7cf7-4362-8f51-95b7b3173965-trusted-ca-bundle\") pod \"console-77dbb5d6f-5t9t6\" (UID: \"4e182135-7cf7-4362-8f51-95b7b3173965\") " pod="openshift-console/console-77dbb5d6f-5t9t6" Oct 30 17:53:34.039606 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:34.039589 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e182135-7cf7-4362-8f51-95b7b3173965-service-ca\") pod \"console-77dbb5d6f-5t9t6\" (UID: \"4e182135-7cf7-4362-8f51-95b7b3173965\") " pod="openshift-console/console-77dbb5d6f-5t9t6" Oct 30 17:53:34.040730 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:34.040707 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e182135-7cf7-4362-8f51-95b7b3173965-console-serving-cert\") pod \"console-77dbb5d6f-5t9t6\" (UID: \"4e182135-7cf7-4362-8f51-95b7b3173965\") " pod="openshift-console/console-77dbb5d6f-5t9t6" Oct 30 17:53:34.041005 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:34.040989 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e182135-7cf7-4362-8f51-95b7b3173965-console-oauth-config\") pod \"console-77dbb5d6f-5t9t6\" (UID: \"4e182135-7cf7-4362-8f51-95b7b3173965\") " pod="openshift-console/console-77dbb5d6f-5t9t6" Oct 30 17:53:34.061058 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:34.061035 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9hhx\" (UniqueName: \"kubernetes.io/projected/4e182135-7cf7-4362-8f51-95b7b3173965-kube-api-access-c9hhx\") pod \"console-77dbb5d6f-5t9t6\" (UID: \"4e182135-7cf7-4362-8f51-95b7b3173965\") " pod="openshift-console/console-77dbb5d6f-5t9t6" Oct 30 17:53:34.226451 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:34.226417 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77dbb5d6f-5t9t6" Oct 30 17:53:34.544887 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:34.544811 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77dbb5d6f-5t9t6"] Oct 30 17:53:34.548393 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:53:34.548353 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e182135_7cf7_4362_8f51_95b7b3173965.slice/crio-70f1aa5868a89a03222620b2b3bf90fe6a1ac699b4ec4888d434cd92670ab4b0 WatchSource:0}: Error finding container 70f1aa5868a89a03222620b2b3bf90fe6a1ac699b4ec4888d434cd92670ab4b0: Status 404 returned error can't find the container with id 70f1aa5868a89a03222620b2b3bf90fe6a1ac699b4ec4888d434cd92670ab4b0 Oct 30 17:53:35.047642 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:35.047602 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77dbb5d6f-5t9t6" event={"ID":"4e182135-7cf7-4362-8f51-95b7b3173965","Type":"ContainerStarted","Data":"8e88f24c20f487267c02fa30cf5e68ed14f24639462bcd2d9b3802b63385b145"} Oct 30 17:53:35.048049 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:35.047663 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77dbb5d6f-5t9t6" event={"ID":"4e182135-7cf7-4362-8f51-95b7b3173965","Type":"ContainerStarted","Data":"70f1aa5868a89a03222620b2b3bf90fe6a1ac699b4ec4888d434cd92670ab4b0"} Oct 30 17:53:35.089087 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:35.089039 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-77dbb5d6f-5t9t6" podStartSLOduration=2.08902645 podStartE2EDuration="2.08902645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:53:35.08863757 +0000 UTC m=+67.576274604" watchObservedRunningTime="2026-10-30 17:53:35.08902645 +0000 UTC m=+67.576663483" Oct 30 17:53:36.052486 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:36.052452 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7df77857d7-shvsp" podUID="4651f7fa-6227-4a0b-a628-76bde7e15a8b" containerName="console" containerID="cri-o://30028aa3f89baa56dbd75e198e0fdf3e87625ae8fd87a59cf9b8f25ba44e2e93" gracePeriod=15 Oct 30 17:53:36.327371 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:36.327301 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7df77857d7-shvsp" Oct 30 17:53:36.358398 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:36.358372 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4651f7fa-6227-4a0b-a628-76bde7e15a8b-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4651f7fa-6227-4a0b-a628-76bde7e15a8b-tmp\") pod \"4651f7fa-6227-4a0b-a628-76bde7e15a8b\" (UID: \"4651f7fa-6227-4a0b-a628-76bde7e15a8b\") " Oct 30 17:53:36.358552 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:36.358404 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4651f7fa-6227-4a0b-a628-76bde7e15a8b-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4651f7fa-6227-4a0b-a628-76bde7e15a8b-console-serving-cert\") pod \"4651f7fa-6227-4a0b-a628-76bde7e15a8b\" (UID: \"4651f7fa-6227-4a0b-a628-76bde7e15a8b\") " Oct 30 17:53:36.358552 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:36.358431 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4651f7fa-6227-4a0b-a628-76bde7e15a8b-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4651f7fa-6227-4a0b-a628-76bde7e15a8b-oauth-serving-cert\") pod \"4651f7fa-6227-4a0b-a628-76bde7e15a8b\" (UID: \"4651f7fa-6227-4a0b-a628-76bde7e15a8b\") " Oct 30 17:53:36.358552 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:36.358467 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4651f7fa-6227-4a0b-a628-76bde7e15a8b-service-ca\" (UniqueName: \"kubernetes.io/configmap/4651f7fa-6227-4a0b-a628-76bde7e15a8b-service-ca\") pod \"4651f7fa-6227-4a0b-a628-76bde7e15a8b\" (UID: \"4651f7fa-6227-4a0b-a628-76bde7e15a8b\") " Oct 30 17:53:36.358552 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:36.358505 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4651f7fa-6227-4a0b-a628-76bde7e15a8b-kube-api-access-khq4x\" (UniqueName: \"kubernetes.io/projected/4651f7fa-6227-4a0b-a628-76bde7e15a8b-kube-api-access-khq4x\") pod \"4651f7fa-6227-4a0b-a628-76bde7e15a8b\" (UID: \"4651f7fa-6227-4a0b-a628-76bde7e15a8b\") " Oct 30 17:53:36.358552 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:36.358534 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4651f7fa-6227-4a0b-a628-76bde7e15a8b-console-config\" (UniqueName: \"kubernetes.io/configmap/4651f7fa-6227-4a0b-a628-76bde7e15a8b-console-config\") pod \"4651f7fa-6227-4a0b-a628-76bde7e15a8b\" (UID: \"4651f7fa-6227-4a0b-a628-76bde7e15a8b\") " Oct 30 17:53:36.358890 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:36.358560 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4651f7fa-6227-4a0b-a628-76bde7e15a8b-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4651f7fa-6227-4a0b-a628-76bde7e15a8b-console-oauth-config\") pod \"4651f7fa-6227-4a0b-a628-76bde7e15a8b\" (UID: \"4651f7fa-6227-4a0b-a628-76bde7e15a8b\") " Oct 30 17:53:36.358890 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:36.358640 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4651f7fa-6227-4a0b-a628-76bde7e15a8b-tmp" pod "4651f7fa-6227-4a0b-a628-76bde7e15a8b" (UID: "4651f7fa-6227-4a0b-a628-76bde7e15a8b"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:53:36.358890 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:36.358849 4455 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4651f7fa-6227-4a0b-a628-76bde7e15a8b-tmp\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:53:36.359041 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:36.358967 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4651f7fa-6227-4a0b-a628-76bde7e15a8b-oauth-serving-cert" pod "4651f7fa-6227-4a0b-a628-76bde7e15a8b" (UID: "4651f7fa-6227-4a0b-a628-76bde7e15a8b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:53:36.359041 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:36.358972 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4651f7fa-6227-4a0b-a628-76bde7e15a8b-console-config" pod "4651f7fa-6227-4a0b-a628-76bde7e15a8b" (UID: "4651f7fa-6227-4a0b-a628-76bde7e15a8b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:53:36.359041 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:36.359025 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4651f7fa-6227-4a0b-a628-76bde7e15a8b-service-ca" pod "4651f7fa-6227-4a0b-a628-76bde7e15a8b" (UID: "4651f7fa-6227-4a0b-a628-76bde7e15a8b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:53:36.360741 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:36.360719 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4651f7fa-6227-4a0b-a628-76bde7e15a8b-console-serving-cert" pod "4651f7fa-6227-4a0b-a628-76bde7e15a8b" (UID: "4651f7fa-6227-4a0b-a628-76bde7e15a8b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:53:36.361415 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:36.361393 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4651f7fa-6227-4a0b-a628-76bde7e15a8b-console-oauth-config" pod "4651f7fa-6227-4a0b-a628-76bde7e15a8b" (UID: "4651f7fa-6227-4a0b-a628-76bde7e15a8b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:53:36.361518 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:36.361434 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4651f7fa-6227-4a0b-a628-76bde7e15a8b-kube-api-access-khq4x" pod "4651f7fa-6227-4a0b-a628-76bde7e15a8b" (UID: "4651f7fa-6227-4a0b-a628-76bde7e15a8b"). InnerVolumeSpecName "kube-api-access-khq4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:53:36.459854 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:36.459818 4455 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4651f7fa-6227-4a0b-a628-76bde7e15a8b-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:53:36.459854 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:36.459850 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khq4x\" (UniqueName: \"kubernetes.io/projected/4651f7fa-6227-4a0b-a628-76bde7e15a8b-kube-api-access-khq4x\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:53:36.459854 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:36.459861 4455 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4651f7fa-6227-4a0b-a628-76bde7e15a8b-console-config\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:53:36.460072 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:36.459869 4455 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4651f7fa-6227-4a0b-a628-76bde7e15a8b-console-oauth-config\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:53:36.460072 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:36.459878 4455 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4651f7fa-6227-4a0b-a628-76bde7e15a8b-console-serving-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:53:36.460072 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:36.459886 4455 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4651f7fa-6227-4a0b-a628-76bde7e15a8b-oauth-serving-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:53:36.804907 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:36.804871 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 17:53:36.805227 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:36.805185 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="efb931b6-9fc2-443a-ab5e-57e003cc56d3" containerName="alertmanager" containerID="cri-o://31e12d734fb8492037f5324ec732b0d744dfb30ab8cb32f403349e0c20348570" gracePeriod=120 Oct 30 17:53:36.805367 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:36.805266 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="efb931b6-9fc2-443a-ab5e-57e003cc56d3" containerName="kube-rbac-proxy-web" containerID="cri-o://bf5d65c13d922359caef794353cd693c50ba97b4fa42d0c5c3e0e028ccaef72d" gracePeriod=120 Oct 30 17:53:36.805367 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:36.805303 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="efb931b6-9fc2-443a-ab5e-57e003cc56d3" containerName="prom-label-proxy" containerID="cri-o://22a74892463a1b031960f163705346a1d23615ad435300c572b654a0692ac1df" gracePeriod=120 Oct 30 17:53:36.805480 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:36.805344 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="efb931b6-9fc2-443a-ab5e-57e003cc56d3" containerName="kube-rbac-proxy" containerID="cri-o://b5ddf634ff70725c2cf974adf92251dc198ec7abfdc2035f99af6c3eb3aacef3" gracePeriod=120 Oct 30 17:53:36.805480 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:36.805256 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="efb931b6-9fc2-443a-ab5e-57e003cc56d3" containerName="kube-rbac-proxy-metric" containerID="cri-o://a878b23685252e9b053dd3da352dd9720ec2b6d74f46c3c0b269c6bfe2983a9e" gracePeriod=120 Oct 30 17:53:36.805480 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:36.805285 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="efb931b6-9fc2-443a-ab5e-57e003cc56d3" containerName="config-reloader" containerID="cri-o://0f8cb3eacd62f7f0094c805da392ec86d6ca3e32e3d2cc628c596eee1ccbab15" gracePeriod=120 Oct 30 17:53:37.058761 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:37.058674 4455 generic.go:356] "Generic (PLEG): container finished" podID="4651f7fa-6227-4a0b-a628-76bde7e15a8b" containerID="30028aa3f89baa56dbd75e198e0fdf3e87625ae8fd87a59cf9b8f25ba44e2e93" exitCode=0 Oct 30 17:53:37.058761 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:37.058747 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7df77857d7-shvsp" Oct 30 17:53:37.059286 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:37.058768 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7df77857d7-shvsp" event={"ID":"4651f7fa-6227-4a0b-a628-76bde7e15a8b","Type":"ContainerDied","Data":"30028aa3f89baa56dbd75e198e0fdf3e87625ae8fd87a59cf9b8f25ba44e2e93"} Oct 30 17:53:37.059286 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:37.058826 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7df77857d7-shvsp" event={"ID":"4651f7fa-6227-4a0b-a628-76bde7e15a8b","Type":"ContainerDied","Data":"5014a9f257f4ff3a6a44f131c9b03cea745cb066ada40cf901e3117e85a0f422"} Oct 30 17:53:37.059286 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:37.058850 4455 scope.go:122] "RemoveContainer" containerID="30028aa3f89baa56dbd75e198e0fdf3e87625ae8fd87a59cf9b8f25ba44e2e93" Oct 30 17:53:37.062507 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:37.062479 4455 generic.go:356] "Generic (PLEG): container finished" podID="efb931b6-9fc2-443a-ab5e-57e003cc56d3" containerID="22a74892463a1b031960f163705346a1d23615ad435300c572b654a0692ac1df" exitCode=0 Oct 30 17:53:37.062607 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:37.062511 4455 generic.go:356] "Generic (PLEG): container finished" podID="efb931b6-9fc2-443a-ab5e-57e003cc56d3" containerID="a878b23685252e9b053dd3da352dd9720ec2b6d74f46c3c0b269c6bfe2983a9e" exitCode=0 Oct 30 17:53:37.062607 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:37.062522 4455 generic.go:356] "Generic (PLEG): container finished" podID="efb931b6-9fc2-443a-ab5e-57e003cc56d3" containerID="b5ddf634ff70725c2cf974adf92251dc198ec7abfdc2035f99af6c3eb3aacef3" exitCode=0 Oct 30 17:53:37.062607 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:37.062531 4455 generic.go:356] "Generic (PLEG): container finished" podID="efb931b6-9fc2-443a-ab5e-57e003cc56d3" containerID="0f8cb3eacd62f7f0094c805da392ec86d6ca3e32e3d2cc628c596eee1ccbab15" exitCode=0 Oct 30 17:53:37.062607 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:37.062543 4455 generic.go:356] "Generic (PLEG): container finished" podID="efb931b6-9fc2-443a-ab5e-57e003cc56d3" containerID="31e12d734fb8492037f5324ec732b0d744dfb30ab8cb32f403349e0c20348570" exitCode=0 Oct 30 17:53:37.062607 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:37.062554 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efb931b6-9fc2-443a-ab5e-57e003cc56d3","Type":"ContainerDied","Data":"22a74892463a1b031960f163705346a1d23615ad435300c572b654a0692ac1df"} Oct 30 17:53:37.062754 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:37.062635 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efb931b6-9fc2-443a-ab5e-57e003cc56d3","Type":"ContainerDied","Data":"a878b23685252e9b053dd3da352dd9720ec2b6d74f46c3c0b269c6bfe2983a9e"} Oct 30 17:53:37.062754 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:37.062678 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efb931b6-9fc2-443a-ab5e-57e003cc56d3","Type":"ContainerDied","Data":"b5ddf634ff70725c2cf974adf92251dc198ec7abfdc2035f99af6c3eb3aacef3"} Oct 30 17:53:37.062754 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:37.062720 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efb931b6-9fc2-443a-ab5e-57e003cc56d3","Type":"ContainerDied","Data":"0f8cb3eacd62f7f0094c805da392ec86d6ca3e32e3d2cc628c596eee1ccbab15"} Oct 30 17:53:37.062845 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:37.062758 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efb931b6-9fc2-443a-ab5e-57e003cc56d3","Type":"ContainerDied","Data":"31e12d734fb8492037f5324ec732b0d744dfb30ab8cb32f403349e0c20348570"} Oct 30 17:53:37.085787 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:37.085766 4455 scope.go:122] "RemoveContainer" containerID="30028aa3f89baa56dbd75e198e0fdf3e87625ae8fd87a59cf9b8f25ba44e2e93" Oct 30 17:53:37.086089 ip-10-0-130-84 kubenswrapper[4455]: E1030 17:53:37.086061 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30028aa3f89baa56dbd75e198e0fdf3e87625ae8fd87a59cf9b8f25ba44e2e93\": container with ID starting with 30028aa3f89baa56dbd75e198e0fdf3e87625ae8fd87a59cf9b8f25ba44e2e93 not found: ID does not exist" containerID="30028aa3f89baa56dbd75e198e0fdf3e87625ae8fd87a59cf9b8f25ba44e2e93" Oct 30 17:53:37.086147 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:37.086099 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30028aa3f89baa56dbd75e198e0fdf3e87625ae8fd87a59cf9b8f25ba44e2e93"} err="failed to get container status \"30028aa3f89baa56dbd75e198e0fdf3e87625ae8fd87a59cf9b8f25ba44e2e93\": rpc error: code = NotFound desc = could not find container \"30028aa3f89baa56dbd75e198e0fdf3e87625ae8fd87a59cf9b8f25ba44e2e93\": container with ID starting with 30028aa3f89baa56dbd75e198e0fdf3e87625ae8fd87a59cf9b8f25ba44e2e93 not found: ID does not exist" Oct 30 17:53:37.106783 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:37.106759 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7df77857d7-shvsp"] Oct 30 17:53:37.110898 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:37.110880 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7df77857d7-shvsp"] Oct 30 17:53:37.630843 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:37.630810 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4651f7fa-6227-4a0b-a628-76bde7e15a8b" path="/var/lib/kubelet/pods/4651f7fa-6227-4a0b-a628-76bde7e15a8b/volumes" Oct 30 17:53:38.069610 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.069583 4455 generic.go:356] "Generic (PLEG): container finished" podID="efb931b6-9fc2-443a-ab5e-57e003cc56d3" containerID="bf5d65c13d922359caef794353cd693c50ba97b4fa42d0c5c3e0e028ccaef72d" exitCode=0 Oct 30 17:53:38.069925 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.069654 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efb931b6-9fc2-443a-ab5e-57e003cc56d3","Type":"ContainerDied","Data":"bf5d65c13d922359caef794353cd693c50ba97b4fa42d0c5c3e0e028ccaef72d"} Oct 30 17:53:38.069925 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.069711 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efb931b6-9fc2-443a-ab5e-57e003cc56d3","Type":"ContainerDied","Data":"847038e5b0db03dd90f5b80801b53f77786863bca99065d027187199e7621d23"} Oct 30 17:53:38.069925 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.069739 4455 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="847038e5b0db03dd90f5b80801b53f77786863bca99065d027187199e7621d23" Oct 30 17:53:38.082148 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.082125 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:38.175248 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.173811 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efb931b6-9fc2-443a-ab5e-57e003cc56d3-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efb931b6-9fc2-443a-ab5e-57e003cc56d3-metrics-client-ca\") pod \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " Oct 30 17:53:38.175248 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.173860 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efb931b6-9fc2-443a-ab5e-57e003cc56d3-tls-assets\" (UniqueName: \"kubernetes.io/projected/efb931b6-9fc2-443a-ab5e-57e003cc56d3-tls-assets\") pod \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " Oct 30 17:53:38.175248 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.173890 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-secret-alertmanager-kube-rbac-proxy\") pod \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " Oct 30 17:53:38.175248 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.173923 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efb931b6-9fc2-443a-ab5e-57e003cc56d3-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efb931b6-9fc2-443a-ab5e-57e003cc56d3-alertmanager-trusted-ca-bundle\") pod \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " Oct 30 17:53:38.175248 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.173963 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-config-volume\" (UniqueName: \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-config-volume\") pod \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " Oct 30 17:53:38.175248 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.174002 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-secret-alertmanager-main-tls\") pod \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " Oct 30 17:53:38.175248 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.174033 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-cluster-tls-config\") pod \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " Oct 30 17:53:38.175248 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.174067 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " Oct 30 17:53:38.175248 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.174125 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-secret-alertmanager-kube-rbac-proxy-web\") pod \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " Oct 30 17:53:38.175248 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.174167 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efb931b6-9fc2-443a-ab5e-57e003cc56d3-kube-api-access-9km54\" (UniqueName: \"kubernetes.io/projected/efb931b6-9fc2-443a-ab5e-57e003cc56d3-kube-api-access-9km54\") pod \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " Oct 30 17:53:38.175248 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.174196 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-web-config\" (UniqueName: \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-web-config\") pod \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " Oct 30 17:53:38.175248 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.174234 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb931b6-9fc2-443a-ab5e-57e003cc56d3-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/efb931b6-9fc2-443a-ab5e-57e003cc56d3-alertmanager-main-db\") pod \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " Oct 30 17:53:38.175248 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.174264 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb931b6-9fc2-443a-ab5e-57e003cc56d3-config-out\" (UniqueName: \"kubernetes.io/empty-dir/efb931b6-9fc2-443a-ab5e-57e003cc56d3-config-out\") pod \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\" (UID: \"efb931b6-9fc2-443a-ab5e-57e003cc56d3\") " Oct 30 17:53:38.176944 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.176877 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efb931b6-9fc2-443a-ab5e-57e003cc56d3-metrics-client-ca" pod "efb931b6-9fc2-443a-ab5e-57e003cc56d3" (UID: "efb931b6-9fc2-443a-ab5e-57e003cc56d3"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:53:38.177068 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.176987 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efb931b6-9fc2-443a-ab5e-57e003cc56d3-alertmanager-trusted-ca-bundle" pod "efb931b6-9fc2-443a-ab5e-57e003cc56d3" (UID: "efb931b6-9fc2-443a-ab5e-57e003cc56d3"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:53:38.180121 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.179935 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-secret-alertmanager-main-tls" pod "efb931b6-9fc2-443a-ab5e-57e003cc56d3" (UID: "efb931b6-9fc2-443a-ab5e-57e003cc56d3"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:53:38.180441 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.180393 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb931b6-9fc2-443a-ab5e-57e003cc56d3-alertmanager-main-db" pod "efb931b6-9fc2-443a-ab5e-57e003cc56d3" (UID: "efb931b6-9fc2-443a-ab5e-57e003cc56d3"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:53:38.181131 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.180947 4455 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efb931b6-9fc2-443a-ab5e-57e003cc56d3-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:53:38.181131 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.180977 4455 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-secret-alertmanager-main-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:53:38.181131 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.181001 4455 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/efb931b6-9fc2-443a-ab5e-57e003cc56d3-alertmanager-main-db\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:53:38.181131 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.181016 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efb931b6-9fc2-443a-ab5e-57e003cc56d3-metrics-client-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:53:38.191801 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.191663 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-secret-alertmanager-kube-rbac-proxy-web" pod "efb931b6-9fc2-443a-ab5e-57e003cc56d3" (UID: "efb931b6-9fc2-443a-ab5e-57e003cc56d3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:53:38.192106 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.192081 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb931b6-9fc2-443a-ab5e-57e003cc56d3-kube-api-access-9km54" pod "efb931b6-9fc2-443a-ab5e-57e003cc56d3" (UID: "efb931b6-9fc2-443a-ab5e-57e003cc56d3"). InnerVolumeSpecName "kube-api-access-9km54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:53:38.192254 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.192228 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-secret-alertmanager-kube-rbac-proxy" pod "efb931b6-9fc2-443a-ab5e-57e003cc56d3" (UID: "efb931b6-9fc2-443a-ab5e-57e003cc56d3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:53:38.192354 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.192258 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb931b6-9fc2-443a-ab5e-57e003cc56d3-tls-assets" pod "efb931b6-9fc2-443a-ab5e-57e003cc56d3" (UID: "efb931b6-9fc2-443a-ab5e-57e003cc56d3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:53:38.192625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.192596 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-secret-alertmanager-kube-rbac-proxy-metric" pod "efb931b6-9fc2-443a-ab5e-57e003cc56d3" (UID: "efb931b6-9fc2-443a-ab5e-57e003cc56d3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:53:38.192884 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.192708 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-config-volume" pod "efb931b6-9fc2-443a-ab5e-57e003cc56d3" (UID: "efb931b6-9fc2-443a-ab5e-57e003cc56d3"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:53:38.195549 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.195509 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb931b6-9fc2-443a-ab5e-57e003cc56d3-config-out" pod "efb931b6-9fc2-443a-ab5e-57e003cc56d3" (UID: "efb931b6-9fc2-443a-ab5e-57e003cc56d3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:53:38.203438 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.203413 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-cluster-tls-config" pod "efb931b6-9fc2-443a-ab5e-57e003cc56d3" (UID: "efb931b6-9fc2-443a-ab5e-57e003cc56d3"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:53:38.205947 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.205925 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-web-config" pod "efb931b6-9fc2-443a-ab5e-57e003cc56d3" (UID: "efb931b6-9fc2-443a-ab5e-57e003cc56d3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:53:38.282008 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.281976 4455 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/efb931b6-9fc2-443a-ab5e-57e003cc56d3-config-out\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:53:38.282008 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.282005 4455 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/efb931b6-9fc2-443a-ab5e-57e003cc56d3-tls-assets\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:53:38.282180 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.282016 4455 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:53:38.282180 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.282025 4455 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:53:38.282180 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.282034 4455 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-cluster-tls-config\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:53:38.282180 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.282042 4455 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:53:38.282180 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.282052 4455 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:53:38.282180 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.282061 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9km54\" (UniqueName: \"kubernetes.io/projected/efb931b6-9fc2-443a-ab5e-57e003cc56d3-kube-api-access-9km54\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:53:38.282180 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:38.282070 4455 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/efb931b6-9fc2-443a-ab5e-57e003cc56d3-web-config\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:53:39.073527 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.073496 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.138286 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.138256 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 17:53:39.149619 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.149592 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 17:53:39.178070 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.178039 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 17:53:39.178337 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.178311 4455 cpu_manager.go:414] "containerMap: removing container" podUID="4651f7fa-6227-4a0b-a628-76bde7e15a8b" containerName="console" Oct 30 17:53:39.178383 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.178341 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4651f7fa-6227-4a0b-a628-76bde7e15a8b" containerName="console" Oct 30 17:53:39.178383 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.178351 4455 cpu_manager.go:414] "containerMap: removing container" podUID="efb931b6-9fc2-443a-ab5e-57e003cc56d3" containerName="init-config-reloader" Oct 30 17:53:39.178383 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.178356 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efb931b6-9fc2-443a-ab5e-57e003cc56d3" containerName="init-config-reloader" Oct 30 17:53:39.178383 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.178364 4455 cpu_manager.go:414] "containerMap: removing container" podUID="efb931b6-9fc2-443a-ab5e-57e003cc56d3" containerName="alertmanager" Oct 30 17:53:39.178383 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.178369 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efb931b6-9fc2-443a-ab5e-57e003cc56d3" containerName="alertmanager" Oct 30 17:53:39.178383 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.178377 4455 cpu_manager.go:414] "containerMap: removing container" podUID="efb931b6-9fc2-443a-ab5e-57e003cc56d3" containerName="kube-rbac-proxy-metric" Oct 30 17:53:39.178383 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.178381 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efb931b6-9fc2-443a-ab5e-57e003cc56d3" containerName="kube-rbac-proxy-metric" Oct 30 17:53:39.178630 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.178390 4455 cpu_manager.go:414] "containerMap: removing container" podUID="efb931b6-9fc2-443a-ab5e-57e003cc56d3" containerName="config-reloader" Oct 30 17:53:39.178630 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.178394 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efb931b6-9fc2-443a-ab5e-57e003cc56d3" containerName="config-reloader" Oct 30 17:53:39.178630 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.178402 4455 cpu_manager.go:414] "containerMap: removing container" podUID="efb931b6-9fc2-443a-ab5e-57e003cc56d3" containerName="kube-rbac-proxy-web" Oct 30 17:53:39.178630 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.178407 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efb931b6-9fc2-443a-ab5e-57e003cc56d3" containerName="kube-rbac-proxy-web" Oct 30 17:53:39.178630 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.178414 4455 cpu_manager.go:414] "containerMap: removing container" podUID="efb931b6-9fc2-443a-ab5e-57e003cc56d3" containerName="kube-rbac-proxy" Oct 30 17:53:39.178630 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.178419 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efb931b6-9fc2-443a-ab5e-57e003cc56d3" containerName="kube-rbac-proxy" Oct 30 17:53:39.178630 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.178429 4455 cpu_manager.go:414] "containerMap: removing container" podUID="efb931b6-9fc2-443a-ab5e-57e003cc56d3" containerName="prom-label-proxy" Oct 30 17:53:39.178630 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.178433 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efb931b6-9fc2-443a-ab5e-57e003cc56d3" containerName="prom-label-proxy" Oct 30 17:53:39.178630 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.178512 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="4651f7fa-6227-4a0b-a628-76bde7e15a8b" containerName="console" Oct 30 17:53:39.178630 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.178519 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="efb931b6-9fc2-443a-ab5e-57e003cc56d3" containerName="alertmanager" Oct 30 17:53:39.178630 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.178524 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="efb931b6-9fc2-443a-ab5e-57e003cc56d3" containerName="kube-rbac-proxy-metric" Oct 30 17:53:39.178630 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.178530 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="efb931b6-9fc2-443a-ab5e-57e003cc56d3" containerName="config-reloader" Oct 30 17:53:39.178630 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.178536 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="efb931b6-9fc2-443a-ab5e-57e003cc56d3" containerName="kube-rbac-proxy-web" Oct 30 17:53:39.178630 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.178541 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="efb931b6-9fc2-443a-ab5e-57e003cc56d3" containerName="kube-rbac-proxy" Oct 30 17:53:39.178630 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.178547 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="efb931b6-9fc2-443a-ab5e-57e003cc56d3" containerName="prom-label-proxy" Oct 30 17:53:39.200782 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.200757 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.215732 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.215704 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 17:53:39.215732 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.215702 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 17:53:39.215915 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.215732 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 17:53:39.215915 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.215766 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 17:53:39.216043 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.216015 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 17:53:39.216202 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.216187 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 17:53:39.217508 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.217490 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 17:53:39.226928 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.226906 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 17:53:39.227141 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.227122 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 17:53:39.227241 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.227220 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-xdrrk\"" Oct 30 17:53:39.227497 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.227481 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 17:53:39.289102 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.289076 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d7fa1284-e214-4839-93d7-dd4b30ba4f34-web-config\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.289289 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.289109 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7fa1284-e214-4839-93d7-dd4b30ba4f34-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.289289 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.289130 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d7fa1284-e214-4839-93d7-dd4b30ba4f34-config-volume\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.289289 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.289146 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d7fa1284-e214-4839-93d7-dd4b30ba4f34-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.289289 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.289219 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d7fa1284-e214-4839-93d7-dd4b30ba4f34-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.289289 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.289269 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d7fa1284-e214-4839-93d7-dd4b30ba4f34-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.289532 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.289361 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x49pr\" (UniqueName: \"kubernetes.io/projected/d7fa1284-e214-4839-93d7-dd4b30ba4f34-kube-api-access-x49pr\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.289532 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.289391 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d7fa1284-e214-4839-93d7-dd4b30ba4f34-config-out\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.289532 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.289410 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d7fa1284-e214-4839-93d7-dd4b30ba4f34-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.289532 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.289493 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d7fa1284-e214-4839-93d7-dd4b30ba4f34-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.289532 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.289508 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d7fa1284-e214-4839-93d7-dd4b30ba4f34-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.289532 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.289525 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d7fa1284-e214-4839-93d7-dd4b30ba4f34-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.289730 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.289549 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7fa1284-e214-4839-93d7-dd4b30ba4f34-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.390615 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.390534 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d7fa1284-e214-4839-93d7-dd4b30ba4f34-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.390615 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.390569 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d7fa1284-e214-4839-93d7-dd4b30ba4f34-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.390615 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.390587 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d7fa1284-e214-4839-93d7-dd4b30ba4f34-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.391392 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.390618 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7fa1284-e214-4839-93d7-dd4b30ba4f34-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.391392 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.390687 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d7fa1284-e214-4839-93d7-dd4b30ba4f34-web-config\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.391392 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.390717 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7fa1284-e214-4839-93d7-dd4b30ba4f34-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.391392 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.390749 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d7fa1284-e214-4839-93d7-dd4b30ba4f34-config-volume\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.391392 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.390770 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d7fa1284-e214-4839-93d7-dd4b30ba4f34-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.391392 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.390794 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d7fa1284-e214-4839-93d7-dd4b30ba4f34-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.391392 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.390814 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d7fa1284-e214-4839-93d7-dd4b30ba4f34-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.391392 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.390836 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x49pr\" (UniqueName: \"kubernetes.io/projected/d7fa1284-e214-4839-93d7-dd4b30ba4f34-kube-api-access-x49pr\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.391392 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.390861 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d7fa1284-e214-4839-93d7-dd4b30ba4f34-config-out\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.391392 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.390881 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d7fa1284-e214-4839-93d7-dd4b30ba4f34-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.391392 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.391159 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d7fa1284-e214-4839-93d7-dd4b30ba4f34-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.392312 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.391487 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7fa1284-e214-4839-93d7-dd4b30ba4f34-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.392312 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.392070 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7fa1284-e214-4839-93d7-dd4b30ba4f34-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.394065 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.394024 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d7fa1284-e214-4839-93d7-dd4b30ba4f34-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.394246 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.394218 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d7fa1284-e214-4839-93d7-dd4b30ba4f34-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.394538 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.394513 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d7fa1284-e214-4839-93d7-dd4b30ba4f34-web-config\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.395007 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.394982 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d7fa1284-e214-4839-93d7-dd4b30ba4f34-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.395095 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.394982 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d7fa1284-e214-4839-93d7-dd4b30ba4f34-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.395095 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.395043 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d7fa1284-e214-4839-93d7-dd4b30ba4f34-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.395095 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.395081 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d7fa1284-e214-4839-93d7-dd4b30ba4f34-config-volume\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.395262 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.395221 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d7fa1284-e214-4839-93d7-dd4b30ba4f34-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.395447 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.395431 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d7fa1284-e214-4839-93d7-dd4b30ba4f34-config-out\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.413547 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.413528 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x49pr\" (UniqueName: \"kubernetes.io/projected/d7fa1284-e214-4839-93d7-dd4b30ba4f34-kube-api-access-x49pr\") pod \"alertmanager-main-0\" (UID: \"d7fa1284-e214-4839-93d7-dd4b30ba4f34\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.616449 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.616404 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:53:39.631665 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.631633 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efb931b6-9fc2-443a-ab5e-57e003cc56d3" path="/var/lib/kubelet/pods/efb931b6-9fc2-443a-ab5e-57e003cc56d3/volumes" Oct 30 17:53:39.963148 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:39.963048 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 17:53:39.976170 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:53:39.976140 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7fa1284_e214_4839_93d7_dd4b30ba4f34.slice/crio-a82b25883137ad4300699d791542bb9fa9e428bdde9bf31ec14da37db4f55b19 WatchSource:0}: Error finding container a82b25883137ad4300699d791542bb9fa9e428bdde9bf31ec14da37db4f55b19: Status 404 returned error can't find the container with id a82b25883137ad4300699d791542bb9fa9e428bdde9bf31ec14da37db4f55b19 Oct 30 17:53:40.084434 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:40.084396 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7fa1284-e214-4839-93d7-dd4b30ba4f34","Type":"ContainerStarted","Data":"9e3357e80f5a68272563964a6f3481bab1c66cce5786d29ea36ff769c3ea72ca"} Oct 30 17:53:40.084820 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:40.084523 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7fa1284-e214-4839-93d7-dd4b30ba4f34","Type":"ContainerStarted","Data":"a82b25883137ad4300699d791542bb9fa9e428bdde9bf31ec14da37db4f55b19"} Oct 30 17:53:41.089785 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:41.089748 4455 generic.go:356] "Generic (PLEG): container finished" podID="d7fa1284-e214-4839-93d7-dd4b30ba4f34" containerID="9e3357e80f5a68272563964a6f3481bab1c66cce5786d29ea36ff769c3ea72ca" exitCode=0 Oct 30 17:53:41.090219 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:41.089832 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7fa1284-e214-4839-93d7-dd4b30ba4f34","Type":"ContainerDied","Data":"9e3357e80f5a68272563964a6f3481bab1c66cce5786d29ea36ff769c3ea72ca"} Oct 30 17:53:42.098374 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:42.098340 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7fa1284-e214-4839-93d7-dd4b30ba4f34","Type":"ContainerStarted","Data":"bd0a39c9afd9056767c6af10e40ed9ad93b4b14d7eca46cf1b4eff3d9fda3522"} Oct 30 17:53:42.098797 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:42.098406 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7fa1284-e214-4839-93d7-dd4b30ba4f34","Type":"ContainerStarted","Data":"9e29831fb8d16bba2f05290f8443e0fa57e2df6f9f21686d66f42d3c0510484c"} Oct 30 17:53:42.098797 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:42.098433 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7fa1284-e214-4839-93d7-dd4b30ba4f34","Type":"ContainerStarted","Data":"096ca534cdf1c0186bcfa5f4d34bbad2ab3cc785bd6fdd8e856506848b2b74c5"} Oct 30 17:53:42.098797 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:42.098456 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7fa1284-e214-4839-93d7-dd4b30ba4f34","Type":"ContainerStarted","Data":"668f0f8952218126d56b223dec49d5173e1c0a40ce4a80dc2669b00ded270eab"} Oct 30 17:53:42.098797 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:42.098480 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7fa1284-e214-4839-93d7-dd4b30ba4f34","Type":"ContainerStarted","Data":"b551382f3cc8a1b6bfe6da333e14c460cfe69c4ad2d812c4b83fe1a7e7532166"} Oct 30 17:53:42.098797 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:42.098505 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7fa1284-e214-4839-93d7-dd4b30ba4f34","Type":"ContainerStarted","Data":"a2248d0d50f78b00d4ea064b95205d7cefd5ac51edbd3add0648643de102d6cc"} Oct 30 17:53:42.162266 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:42.161081 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.150431483 podStartE2EDuration="3.16106425s" totalImagesPullingTime="0s" totalInitContainerRuntime="10.632767ms" isStatefulPod=false podCreationTimestamp="2026-10-30 17:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:53:42.160596838 +0000 UTC m=+74.648233897" watchObservedRunningTime="2026-10-30 17:53:42.16106425 +0000 UTC m=+74.648701285" Oct 30 17:53:44.227295 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:44.227253 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77dbb5d6f-5t9t6" Oct 30 17:53:44.227800 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:44.227348 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77dbb5d6f-5t9t6" Oct 30 17:53:44.233387 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:44.233363 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77dbb5d6f-5t9t6" Oct 30 17:53:45.116959 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:45.116930 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77dbb5d6f-5t9t6" Oct 30 17:53:45.189662 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:53:45.189622 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f56c74c7-9xv4s"] Oct 30 17:54:00.383635 ip-10-0-130-84 kubenswrapper[4455]: E1030 17:54:00.383598 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 17:54:10.212255 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:10.212211 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-f56c74c7-9xv4s" podUID="fc1fd6e2-d215-4f2a-9bff-cb363aa8909f" containerName="console" containerID="cri-o://bd07108aaee5af81ae0daf2bf821ed5b8f7490eb19629992f1b85f72fc3f8192" gracePeriod=15 Oct 30 17:54:10.464999 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:10.464950 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f56c74c7-9xv4s" Oct 30 17:54:10.553575 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:10.553535 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-oauth-serving-cert\") pod \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\" (UID: \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\") " Oct 30 17:54:10.553735 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:10.553643 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-tmp\") pod \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\" (UID: \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\") " Oct 30 17:54:10.553735 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:10.553674 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-kube-api-access-j8lgt\" (UniqueName: \"kubernetes.io/projected/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-kube-api-access-j8lgt\") pod \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\" (UID: \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\") " Oct 30 17:54:10.553735 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:10.553695 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-service-ca\") pod \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\" (UID: \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\") " Oct 30 17:54:10.553735 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:10.553714 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-console-oauth-config\") pod \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\" (UID: \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\") " Oct 30 17:54:10.553925 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:10.553749 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-console-config\" (UniqueName: \"kubernetes.io/configmap/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-console-config\") pod \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\" (UID: \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\") " Oct 30 17:54:10.553925 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:10.553785 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-trusted-ca-bundle\") pod \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\" (UID: \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\") " Oct 30 17:54:10.553925 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:10.553825 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-console-serving-cert\") pod \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\" (UID: \"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f\") " Oct 30 17:54:10.554075 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:10.553939 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-tmp" pod "fc1fd6e2-d215-4f2a-9bff-cb363aa8909f" (UID: "fc1fd6e2-d215-4f2a-9bff-cb363aa8909f"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:10.554075 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:10.553952 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-oauth-serving-cert" pod "fc1fd6e2-d215-4f2a-9bff-cb363aa8909f" (UID: "fc1fd6e2-d215-4f2a-9bff-cb363aa8909f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:54:10.554178 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:10.554107 4455 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-oauth-serving-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:54:10.554178 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:10.554147 4455 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-tmp\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:54:10.554300 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:10.554277 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-service-ca" pod "fc1fd6e2-d215-4f2a-9bff-cb363aa8909f" (UID: "fc1fd6e2-d215-4f2a-9bff-cb363aa8909f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:54:10.554419 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:10.554385 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-trusted-ca-bundle" pod "fc1fd6e2-d215-4f2a-9bff-cb363aa8909f" (UID: "fc1fd6e2-d215-4f2a-9bff-cb363aa8909f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:54:10.554419 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:10.554404 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-console-config" pod "fc1fd6e2-d215-4f2a-9bff-cb363aa8909f" (UID: "fc1fd6e2-d215-4f2a-9bff-cb363aa8909f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:54:10.556164 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:10.556137 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-console-oauth-config" pod "fc1fd6e2-d215-4f2a-9bff-cb363aa8909f" (UID: "fc1fd6e2-d215-4f2a-9bff-cb363aa8909f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:54:10.556164 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:10.556152 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-console-serving-cert" pod "fc1fd6e2-d215-4f2a-9bff-cb363aa8909f" (UID: "fc1fd6e2-d215-4f2a-9bff-cb363aa8909f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:54:10.556288 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:10.556261 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-kube-api-access-j8lgt" pod "fc1fd6e2-d215-4f2a-9bff-cb363aa8909f" (UID: "fc1fd6e2-d215-4f2a-9bff-cb363aa8909f"). InnerVolumeSpecName "kube-api-access-j8lgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:54:10.655371 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:10.655313 4455 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-console-config\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:54:10.655371 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:10.655364 4455 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-trusted-ca-bundle\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:54:10.655371 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:10.655379 4455 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-console-serving-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:54:10.655607 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:10.655392 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8lgt\" (UniqueName: \"kubernetes.io/projected/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-kube-api-access-j8lgt\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:54:10.655607 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:10.655403 4455 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:54:10.655607 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:10.655413 4455 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f-console-oauth-config\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:54:11.222575 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:11.222546 4455 generic.go:356] "Generic (PLEG): container finished" podID="fc1fd6e2-d215-4f2a-9bff-cb363aa8909f" containerID="bd07108aaee5af81ae0daf2bf821ed5b8f7490eb19629992f1b85f72fc3f8192" exitCode=0 Oct 30 17:54:11.223090 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:11.222612 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f56c74c7-9xv4s" Oct 30 17:54:11.223090 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:11.222632 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f56c74c7-9xv4s" event={"ID":"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f","Type":"ContainerDied","Data":"bd07108aaee5af81ae0daf2bf821ed5b8f7490eb19629992f1b85f72fc3f8192"} Oct 30 17:54:11.223090 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:11.222682 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f56c74c7-9xv4s" event={"ID":"fc1fd6e2-d215-4f2a-9bff-cb363aa8909f","Type":"ContainerDied","Data":"4c33faac5dfc6fd3a8001a9eab2862623a4fc44f47eb65fa6997dbb897d2394b"} Oct 30 17:54:11.223090 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:11.222700 4455 scope.go:122] "RemoveContainer" containerID="bd07108aaee5af81ae0daf2bf821ed5b8f7490eb19629992f1b85f72fc3f8192" Oct 30 17:54:11.232394 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:11.232377 4455 scope.go:122] "RemoveContainer" containerID="bd07108aaee5af81ae0daf2bf821ed5b8f7490eb19629992f1b85f72fc3f8192" Oct 30 17:54:11.232689 ip-10-0-130-84 kubenswrapper[4455]: E1030 17:54:11.232672 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd07108aaee5af81ae0daf2bf821ed5b8f7490eb19629992f1b85f72fc3f8192\": container with ID starting with bd07108aaee5af81ae0daf2bf821ed5b8f7490eb19629992f1b85f72fc3f8192 not found: ID does not exist" containerID="bd07108aaee5af81ae0daf2bf821ed5b8f7490eb19629992f1b85f72fc3f8192" Oct 30 17:54:11.232735 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:11.232699 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd07108aaee5af81ae0daf2bf821ed5b8f7490eb19629992f1b85f72fc3f8192"} err="failed to get container status \"bd07108aaee5af81ae0daf2bf821ed5b8f7490eb19629992f1b85f72fc3f8192\": rpc error: code = NotFound desc = could not find container \"bd07108aaee5af81ae0daf2bf821ed5b8f7490eb19629992f1b85f72fc3f8192\": container with ID starting with bd07108aaee5af81ae0daf2bf821ed5b8f7490eb19629992f1b85f72fc3f8192 not found: ID does not exist" Oct 30 17:54:11.270232 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:11.270206 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f56c74c7-9xv4s"] Oct 30 17:54:11.281648 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:11.281621 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f56c74c7-9xv4s"] Oct 30 17:54:11.632206 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:54:11.632113 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc1fd6e2-d215-4f2a-9bff-cb363aa8909f" path="/var/lib/kubelet/pods/fc1fd6e2-d215-4f2a-9bff-cb363aa8909f/volumes" Oct 30 17:54:30.382959 ip-10-0-130-84 kubenswrapper[4455]: E1030 17:54:30.382921 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 17:55:00.382972 ip-10-0-130-84 kubenswrapper[4455]: E1030 17:55:00.382881 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 17:55:30.383743 ip-10-0-130-84 kubenswrapper[4455]: E1030 17:55:30.383706 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 17:56:00.383980 ip-10-0-130-84 kubenswrapper[4455]: E1030 17:56:00.383938 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 17:56:05.429265 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:05.429232 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142cefpdp4n"] Oct 30 17:56:05.429691 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:05.429594 4455 cpu_manager.go:414] "containerMap: removing container" podUID="fc1fd6e2-d215-4f2a-9bff-cb363aa8909f" containerName="console" Oct 30 17:56:05.429691 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:05.429606 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc1fd6e2-d215-4f2a-9bff-cb363aa8909f" containerName="console" Oct 30 17:56:05.429691 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:05.429662 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc1fd6e2-d215-4f2a-9bff-cb363aa8909f" containerName="console" Oct 30 17:56:05.431920 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:05.431900 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142cefpdp4n" Oct 30 17:56:05.445893 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:05.445873 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 17:56:05.457443 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:05.457410 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 17:56:05.457640 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:05.457623 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"fdb36248658848c45b2cd30265e4aa38f5747d-4cc5d9d1-dockercfg-rn4l6\"" Oct 30 17:56:05.458695 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:05.458673 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142cefpdp4n"] Oct 30 17:56:05.580051 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:05.580015 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/820f4cbf-c8c9-40f4-bf53-230143511533-bundle\") pod \"fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142cefpdp4n\" (UID: \"820f4cbf-c8c9-40f4-bf53-230143511533\") " pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142cefpdp4n" Oct 30 17:56:05.580222 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:05.580062 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjrtz\" (UniqueName: \"kubernetes.io/projected/820f4cbf-c8c9-40f4-bf53-230143511533-kube-api-access-kjrtz\") pod \"fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142cefpdp4n\" (UID: \"820f4cbf-c8c9-40f4-bf53-230143511533\") " pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142cefpdp4n" Oct 30 17:56:05.580222 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:05.580082 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/820f4cbf-c8c9-40f4-bf53-230143511533-util\") pod \"fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142cefpdp4n\" (UID: \"820f4cbf-c8c9-40f4-bf53-230143511533\") " pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142cefpdp4n" Oct 30 17:56:05.681128 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:05.681034 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/820f4cbf-c8c9-40f4-bf53-230143511533-bundle\") pod \"fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142cefpdp4n\" (UID: \"820f4cbf-c8c9-40f4-bf53-230143511533\") " pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142cefpdp4n" Oct 30 17:56:05.681128 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:05.681080 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjrtz\" (UniqueName: \"kubernetes.io/projected/820f4cbf-c8c9-40f4-bf53-230143511533-kube-api-access-kjrtz\") pod \"fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142cefpdp4n\" (UID: \"820f4cbf-c8c9-40f4-bf53-230143511533\") " pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142cefpdp4n" Oct 30 17:56:05.681128 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:05.681099 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/820f4cbf-c8c9-40f4-bf53-230143511533-util\") pod \"fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142cefpdp4n\" (UID: \"820f4cbf-c8c9-40f4-bf53-230143511533\") " pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142cefpdp4n" Oct 30 17:56:05.681472 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:05.681455 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/820f4cbf-c8c9-40f4-bf53-230143511533-util\") pod \"fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142cefpdp4n\" (UID: \"820f4cbf-c8c9-40f4-bf53-230143511533\") " pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142cefpdp4n" Oct 30 17:56:05.681567 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:05.681547 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/820f4cbf-c8c9-40f4-bf53-230143511533-bundle\") pod \"fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142cefpdp4n\" (UID: \"820f4cbf-c8c9-40f4-bf53-230143511533\") " pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142cefpdp4n" Oct 30 17:56:05.704386 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:05.704349 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjrtz\" (UniqueName: \"kubernetes.io/projected/820f4cbf-c8c9-40f4-bf53-230143511533-kube-api-access-kjrtz\") pod \"fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142cefpdp4n\" (UID: \"820f4cbf-c8c9-40f4-bf53-230143511533\") " pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142cefpdp4n" Oct 30 17:56:05.745200 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:05.745169 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142cefpdp4n" Oct 30 17:56:05.830729 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:05.830694 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644chl46"] Oct 30 17:56:05.834105 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:05.834079 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644chl46" Oct 30 17:56:05.848244 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:05.848214 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-w4jdn\"" Oct 30 17:56:05.856065 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:05.856043 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644chl46"] Oct 30 17:56:05.982642 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:05.982610 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pldts\" (UniqueName: \"kubernetes.io/projected/b6aa9825-1700-4360-9082-42ba116a2727-kube-api-access-pldts\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644chl46\" (UID: \"b6aa9825-1700-4360-9082-42ba116a2727\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644chl46" Oct 30 17:56:05.982808 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:05.982657 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6aa9825-1700-4360-9082-42ba116a2727-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644chl46\" (UID: \"b6aa9825-1700-4360-9082-42ba116a2727\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644chl46" Oct 30 17:56:05.982808 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:05.982777 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6aa9825-1700-4360-9082-42ba116a2727-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644chl46\" (UID: \"b6aa9825-1700-4360-9082-42ba116a2727\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644chl46" Oct 30 17:56:06.039267 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:06.039234 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142cefpdp4n"] Oct 30 17:56:06.042943 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:56:06.042915 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod820f4cbf_c8c9_40f4_bf53_230143511533.slice/crio-fc77e4701814c43a9c39396a8edef9d6e99082b67360494a2b00f9c526955a54 WatchSource:0}: Error finding container fc77e4701814c43a9c39396a8edef9d6e99082b67360494a2b00f9c526955a54: Status 404 returned error can't find the container with id fc77e4701814c43a9c39396a8edef9d6e99082b67360494a2b00f9c526955a54 Oct 30 17:56:06.083851 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:06.083811 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6aa9825-1700-4360-9082-42ba116a2727-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644chl46\" (UID: \"b6aa9825-1700-4360-9082-42ba116a2727\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644chl46" Oct 30 17:56:06.084040 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:06.083900 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6aa9825-1700-4360-9082-42ba116a2727-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644chl46\" (UID: \"b6aa9825-1700-4360-9082-42ba116a2727\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644chl46" Oct 30 17:56:06.084040 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:06.083960 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pldts\" (UniqueName: \"kubernetes.io/projected/b6aa9825-1700-4360-9082-42ba116a2727-kube-api-access-pldts\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644chl46\" (UID: \"b6aa9825-1700-4360-9082-42ba116a2727\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644chl46" Oct 30 17:56:06.084212 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:06.084192 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6aa9825-1700-4360-9082-42ba116a2727-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644chl46\" (UID: \"b6aa9825-1700-4360-9082-42ba116a2727\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644chl46" Oct 30 17:56:06.084275 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:06.084249 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6aa9825-1700-4360-9082-42ba116a2727-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644chl46\" (UID: \"b6aa9825-1700-4360-9082-42ba116a2727\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644chl46" Oct 30 17:56:06.107135 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:06.107096 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pldts\" (UniqueName: \"kubernetes.io/projected/b6aa9825-1700-4360-9082-42ba116a2727-kube-api-access-pldts\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644chl46\" (UID: \"b6aa9825-1700-4360-9082-42ba116a2727\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644chl46" Oct 30 17:56:06.148344 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:06.148284 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644chl46" Oct 30 17:56:06.440874 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:06.440800 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644chl46"] Oct 30 17:56:06.444191 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:56:06.444155 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6aa9825_1700_4360_9082_42ba116a2727.slice/crio-6dfb83ad57e98607d04dcec3b10555af1934a6b5c12d758fd83e1022ef4591c8 WatchSource:0}: Error finding container 6dfb83ad57e98607d04dcec3b10555af1934a6b5c12d758fd83e1022ef4591c8: Status 404 returned error can't find the container with id 6dfb83ad57e98607d04dcec3b10555af1934a6b5c12d758fd83e1022ef4591c8 Oct 30 17:56:06.666776 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:06.666733 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644chl46" event={"ID":"b6aa9825-1700-4360-9082-42ba116a2727","Type":"ContainerStarted","Data":"6dfb83ad57e98607d04dcec3b10555af1934a6b5c12d758fd83e1022ef4591c8"} Oct 30 17:56:06.668069 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:06.668037 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142cefpdp4n" event={"ID":"820f4cbf-c8c9-40f4-bf53-230143511533","Type":"ContainerStarted","Data":"fc77e4701814c43a9c39396a8edef9d6e99082b67360494a2b00f9c526955a54"} Oct 30 17:56:07.819934 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:07.819899 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2"] Oct 30 17:56:07.822611 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:07.822585 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:56:07.837007 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:07.836984 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kj7cx\"" Oct 30 17:56:07.837858 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:07.837780 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 17:56:07.837858 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:07.837815 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 17:56:07.838006 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:07.837895 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 17:56:07.856489 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:07.856458 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2"] Oct 30 17:56:07.967770 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:07.967736 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p"] Oct 30 17:56:07.970380 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:07.970357 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" Oct 30 17:56:07.988009 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:07.987983 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-wxg7k\"" Oct 30 17:56:07.991078 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:07.990986 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p"] Oct 30 17:56:08.003456 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.003417 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d316f7-79bb-4e92-be1e-306552cc08b6-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-h2gf2\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:56:08.004385 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.003462 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d316f7-79bb-4e92-be1e-306552cc08b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-h2gf2\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:56:08.004385 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.003510 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d316f7-79bb-4e92-be1e-306552cc08b6-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-h2gf2\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:56:08.004385 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.003534 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d316f7-79bb-4e92-be1e-306552cc08b6-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-h2gf2\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:56:08.004385 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.003581 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcq94\" (UniqueName: \"kubernetes.io/projected/c0d316f7-79bb-4e92-be1e-306552cc08b6-kube-api-access-lcq94\") pod \"maas-default-gateway-openshift-default-67bf986f99-h2gf2\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:56:08.004385 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.003610 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d316f7-79bb-4e92-be1e-306552cc08b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-h2gf2\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:56:08.004385 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.003628 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d316f7-79bb-4e92-be1e-306552cc08b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-h2gf2\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:56:08.004385 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.003649 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d316f7-79bb-4e92-be1e-306552cc08b6-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-h2gf2\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:56:08.004385 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.003674 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d316f7-79bb-4e92-be1e-306552cc08b6-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-h2gf2\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:56:08.104739 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.104654 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d316f7-79bb-4e92-be1e-306552cc08b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-h2gf2\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:56:08.104739 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.104712 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/877f995b-cecd-4842-ba49-963b5c1f0590-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p\" (UID: \"877f995b-cecd-4842-ba49-963b5c1f0590\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" Oct 30 17:56:08.104936 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.104746 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d316f7-79bb-4e92-be1e-306552cc08b6-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-h2gf2\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:56:08.104936 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.104799 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d316f7-79bb-4e92-be1e-306552cc08b6-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-h2gf2\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:56:08.105186 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.105169 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d316f7-79bb-4e92-be1e-306552cc08b6-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-h2gf2\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:56:08.105186 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.105171 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/877f995b-cecd-4842-ba49-963b5c1f0590-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p\" (UID: \"877f995b-cecd-4842-ba49-963b5c1f0590\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" Oct 30 17:56:08.105309 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.105232 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcq94\" (UniqueName: \"kubernetes.io/projected/c0d316f7-79bb-4e92-be1e-306552cc08b6-kube-api-access-lcq94\") pod \"maas-default-gateway-openshift-default-67bf986f99-h2gf2\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:56:08.105309 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.105290 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnbjj\" (UniqueName: \"kubernetes.io/projected/877f995b-cecd-4842-ba49-963b5c1f0590-kube-api-access-cnbjj\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p\" (UID: \"877f995b-cecd-4842-ba49-963b5c1f0590\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" Oct 30 17:56:08.105459 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.105350 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/877f995b-cecd-4842-ba49-963b5c1f0590-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p\" (UID: \"877f995b-cecd-4842-ba49-963b5c1f0590\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" Oct 30 17:56:08.105518 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.105466 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d316f7-79bb-4e92-be1e-306552cc08b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-h2gf2\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:56:08.105518 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.105510 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d316f7-79bb-4e92-be1e-306552cc08b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-h2gf2\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:56:08.105623 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.105542 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/877f995b-cecd-4842-ba49-963b5c1f0590-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p\" (UID: \"877f995b-cecd-4842-ba49-963b5c1f0590\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" Oct 30 17:56:08.105623 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.105575 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d316f7-79bb-4e92-be1e-306552cc08b6-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-h2gf2\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:56:08.105623 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.105601 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d316f7-79bb-4e92-be1e-306552cc08b6-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-h2gf2\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:56:08.105775 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.105627 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/877f995b-cecd-4842-ba49-963b5c1f0590-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p\" (UID: \"877f995b-cecd-4842-ba49-963b5c1f0590\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" Oct 30 17:56:08.105775 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.105666 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/877f995b-cecd-4842-ba49-963b5c1f0590-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p\" (UID: \"877f995b-cecd-4842-ba49-963b5c1f0590\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" Oct 30 17:56:08.105775 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.105709 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/877f995b-cecd-4842-ba49-963b5c1f0590-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p\" (UID: \"877f995b-cecd-4842-ba49-963b5c1f0590\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" Oct 30 17:56:08.105775 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.105747 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/877f995b-cecd-4842-ba49-963b5c1f0590-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p\" (UID: \"877f995b-cecd-4842-ba49-963b5c1f0590\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" Oct 30 17:56:08.105960 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.105775 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d316f7-79bb-4e92-be1e-306552cc08b6-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-h2gf2\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:56:08.106019 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.105993 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d316f7-79bb-4e92-be1e-306552cc08b6-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-h2gf2\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:56:08.106164 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.106140 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d316f7-79bb-4e92-be1e-306552cc08b6-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-h2gf2\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:56:08.106258 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.106190 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d316f7-79bb-4e92-be1e-306552cc08b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-h2gf2\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:56:08.106314 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.106280 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d316f7-79bb-4e92-be1e-306552cc08b6-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-h2gf2\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:56:08.107842 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.107797 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d316f7-79bb-4e92-be1e-306552cc08b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-h2gf2\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:56:08.108388 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.108362 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d316f7-79bb-4e92-be1e-306552cc08b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-h2gf2\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:56:08.130351 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.128692 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d316f7-79bb-4e92-be1e-306552cc08b6-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-h2gf2\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:56:08.130351 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.128817 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcq94\" (UniqueName: \"kubernetes.io/projected/c0d316f7-79bb-4e92-be1e-306552cc08b6-kube-api-access-lcq94\") pod \"maas-default-gateway-openshift-default-67bf986f99-h2gf2\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:56:08.139604 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.139574 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:56:08.206339 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.206284 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/877f995b-cecd-4842-ba49-963b5c1f0590-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p\" (UID: \"877f995b-cecd-4842-ba49-963b5c1f0590\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" Oct 30 17:56:08.206465 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.206406 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/877f995b-cecd-4842-ba49-963b5c1f0590-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p\" (UID: \"877f995b-cecd-4842-ba49-963b5c1f0590\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" Oct 30 17:56:08.206465 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.206454 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnbjj\" (UniqueName: \"kubernetes.io/projected/877f995b-cecd-4842-ba49-963b5c1f0590-kube-api-access-cnbjj\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p\" (UID: \"877f995b-cecd-4842-ba49-963b5c1f0590\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" Oct 30 17:56:08.206580 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.206485 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/877f995b-cecd-4842-ba49-963b5c1f0590-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p\" (UID: \"877f995b-cecd-4842-ba49-963b5c1f0590\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" Oct 30 17:56:08.206580 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.206519 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/877f995b-cecd-4842-ba49-963b5c1f0590-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p\" (UID: \"877f995b-cecd-4842-ba49-963b5c1f0590\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" Oct 30 17:56:08.206580 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.206547 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/877f995b-cecd-4842-ba49-963b5c1f0590-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p\" (UID: \"877f995b-cecd-4842-ba49-963b5c1f0590\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" Oct 30 17:56:08.206580 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.206577 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/877f995b-cecd-4842-ba49-963b5c1f0590-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p\" (UID: \"877f995b-cecd-4842-ba49-963b5c1f0590\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" Oct 30 17:56:08.206784 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.206613 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/877f995b-cecd-4842-ba49-963b5c1f0590-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p\" (UID: \"877f995b-cecd-4842-ba49-963b5c1f0590\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" Oct 30 17:56:08.206784 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.206649 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/877f995b-cecd-4842-ba49-963b5c1f0590-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p\" (UID: \"877f995b-cecd-4842-ba49-963b5c1f0590\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" Oct 30 17:56:08.207084 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.207057 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/877f995b-cecd-4842-ba49-963b5c1f0590-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p\" (UID: \"877f995b-cecd-4842-ba49-963b5c1f0590\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" Oct 30 17:56:08.207767 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.207744 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/877f995b-cecd-4842-ba49-963b5c1f0590-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p\" (UID: \"877f995b-cecd-4842-ba49-963b5c1f0590\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" Oct 30 17:56:08.208039 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.207975 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/877f995b-cecd-4842-ba49-963b5c1f0590-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p\" (UID: \"877f995b-cecd-4842-ba49-963b5c1f0590\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" Oct 30 17:56:08.208245 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.208217 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/877f995b-cecd-4842-ba49-963b5c1f0590-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p\" (UID: \"877f995b-cecd-4842-ba49-963b5c1f0590\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" Oct 30 17:56:08.208904 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.208835 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/877f995b-cecd-4842-ba49-963b5c1f0590-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p\" (UID: \"877f995b-cecd-4842-ba49-963b5c1f0590\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" Oct 30 17:56:08.211812 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.211761 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/877f995b-cecd-4842-ba49-963b5c1f0590-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p\" (UID: \"877f995b-cecd-4842-ba49-963b5c1f0590\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" Oct 30 17:56:08.213365 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.213343 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/877f995b-cecd-4842-ba49-963b5c1f0590-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p\" (UID: \"877f995b-cecd-4842-ba49-963b5c1f0590\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" Oct 30 17:56:08.232144 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.232100 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnbjj\" (UniqueName: \"kubernetes.io/projected/877f995b-cecd-4842-ba49-963b5c1f0590-kube-api-access-cnbjj\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p\" (UID: \"877f995b-cecd-4842-ba49-963b5c1f0590\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" Oct 30 17:56:08.232697 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.232653 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/877f995b-cecd-4842-ba49-963b5c1f0590-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p\" (UID: \"877f995b-cecd-4842-ba49-963b5c1f0590\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" Oct 30 17:56:08.293922 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.293872 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" Oct 30 17:56:08.460058 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.459926 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2"] Oct 30 17:56:08.464506 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:56:08.464432 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d316f7_79bb_4e92_be1e_306552cc08b6.slice/crio-61cdefb0c85be7bc51237fad896e73094c0156dc9a66fee1ed25740e7ba168fe WatchSource:0}: Error finding container 61cdefb0c85be7bc51237fad896e73094c0156dc9a66fee1ed25740e7ba168fe: Status 404 returned error can't find the container with id 61cdefb0c85be7bc51237fad896e73094c0156dc9a66fee1ed25740e7ba168fe Oct 30 17:56:08.607743 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.607702 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p"] Oct 30 17:56:08.612476 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:56:08.612433 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod877f995b_cecd_4842_ba49_963b5c1f0590.slice/crio-e7f360dbfa00942e8c21431d81730707c4d5d80e2d34195d023857a9d59cde8e WatchSource:0}: Error finding container e7f360dbfa00942e8c21431d81730707c4d5d80e2d34195d023857a9d59cde8e: Status 404 returned error can't find the container with id e7f360dbfa00942e8c21431d81730707c4d5d80e2d34195d023857a9d59cde8e Oct 30 17:56:08.679995 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.679919 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" event={"ID":"c0d316f7-79bb-4e92-be1e-306552cc08b6","Type":"ContainerStarted","Data":"61cdefb0c85be7bc51237fad896e73094c0156dc9a66fee1ed25740e7ba168fe"} Oct 30 17:56:08.681382 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:08.681350 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" event={"ID":"877f995b-cecd-4842-ba49-963b5c1f0590","Type":"ContainerStarted","Data":"e7f360dbfa00942e8c21431d81730707c4d5d80e2d34195d023857a9d59cde8e"} Oct 30 17:56:09.246985 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:09.246760 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4dzmb"] Oct 30 17:56:09.252797 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:09.252589 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4dzmb" Oct 30 17:56:09.279431 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:09.279307 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4dzmb"] Oct 30 17:56:09.302189 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:09.302155 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-2ncdz\"" Oct 30 17:56:09.419183 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:09.418817 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4da73a77-4556-4d95-8e25-5eb2d1c7dc6b-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4dzmb\" (UID: \"4da73a77-4556-4d95-8e25-5eb2d1c7dc6b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4dzmb" Oct 30 17:56:09.419183 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:09.418870 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92lh9\" (UniqueName: \"kubernetes.io/projected/4da73a77-4556-4d95-8e25-5eb2d1c7dc6b-kube-api-access-92lh9\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4dzmb\" (UID: \"4da73a77-4556-4d95-8e25-5eb2d1c7dc6b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4dzmb" Oct 30 17:56:09.419183 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:09.418938 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4da73a77-4556-4d95-8e25-5eb2d1c7dc6b-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4dzmb\" (UID: \"4da73a77-4556-4d95-8e25-5eb2d1c7dc6b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4dzmb" Oct 30 17:56:09.520407 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:09.520211 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4da73a77-4556-4d95-8e25-5eb2d1c7dc6b-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4dzmb\" (UID: \"4da73a77-4556-4d95-8e25-5eb2d1c7dc6b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4dzmb" Oct 30 17:56:09.520407 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:09.520258 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92lh9\" (UniqueName: \"kubernetes.io/projected/4da73a77-4556-4d95-8e25-5eb2d1c7dc6b-kube-api-access-92lh9\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4dzmb\" (UID: \"4da73a77-4556-4d95-8e25-5eb2d1c7dc6b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4dzmb" Oct 30 17:56:09.520407 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:09.520313 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4da73a77-4556-4d95-8e25-5eb2d1c7dc6b-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4dzmb\" (UID: \"4da73a77-4556-4d95-8e25-5eb2d1c7dc6b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4dzmb" Oct 30 17:56:09.520955 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:09.520730 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4da73a77-4556-4d95-8e25-5eb2d1c7dc6b-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4dzmb\" (UID: \"4da73a77-4556-4d95-8e25-5eb2d1c7dc6b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4dzmb" Oct 30 17:56:09.520955 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:09.520911 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4da73a77-4556-4d95-8e25-5eb2d1c7dc6b-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4dzmb\" (UID: \"4da73a77-4556-4d95-8e25-5eb2d1c7dc6b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4dzmb" Oct 30 17:56:09.548163 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:09.548097 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92lh9\" (UniqueName: \"kubernetes.io/projected/4da73a77-4556-4d95-8e25-5eb2d1c7dc6b-kube-api-access-92lh9\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4dzmb\" (UID: \"4da73a77-4556-4d95-8e25-5eb2d1c7dc6b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4dzmb" Oct 30 17:56:09.571024 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:09.570766 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4dzmb" Oct 30 17:56:09.603517 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:09.602519 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph4lw"] Oct 30 17:56:09.606779 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:09.606666 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph4lw" Oct 30 17:56:09.622292 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:09.622262 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-7jmzb\"" Oct 30 17:56:09.623264 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:09.623236 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph4lw"] Oct 30 17:56:09.722708 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:09.722442 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be2d95d2-5a8b-410e-9ced-2e7e3a3461ec-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph4lw\" (UID: \"be2d95d2-5a8b-410e-9ced-2e7e3a3461ec\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph4lw" Oct 30 17:56:09.722708 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:09.722510 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be2d95d2-5a8b-410e-9ced-2e7e3a3461ec-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph4lw\" (UID: \"be2d95d2-5a8b-410e-9ced-2e7e3a3461ec\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph4lw" Oct 30 17:56:09.722708 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:09.722542 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv82k\" (UniqueName: \"kubernetes.io/projected/be2d95d2-5a8b-410e-9ced-2e7e3a3461ec-kube-api-access-fv82k\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph4lw\" (UID: \"be2d95d2-5a8b-410e-9ced-2e7e3a3461ec\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph4lw" Oct 30 17:56:09.823368 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:09.823275 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be2d95d2-5a8b-410e-9ced-2e7e3a3461ec-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph4lw\" (UID: \"be2d95d2-5a8b-410e-9ced-2e7e3a3461ec\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph4lw" Oct 30 17:56:09.823368 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:09.823353 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv82k\" (UniqueName: \"kubernetes.io/projected/be2d95d2-5a8b-410e-9ced-2e7e3a3461ec-kube-api-access-fv82k\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph4lw\" (UID: \"be2d95d2-5a8b-410e-9ced-2e7e3a3461ec\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph4lw" Oct 30 17:56:09.823576 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:09.823489 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be2d95d2-5a8b-410e-9ced-2e7e3a3461ec-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph4lw\" (UID: \"be2d95d2-5a8b-410e-9ced-2e7e3a3461ec\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph4lw" Oct 30 17:56:09.823908 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:09.823882 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be2d95d2-5a8b-410e-9ced-2e7e3a3461ec-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph4lw\" (UID: \"be2d95d2-5a8b-410e-9ced-2e7e3a3461ec\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph4lw" Oct 30 17:56:09.824492 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:09.824470 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be2d95d2-5a8b-410e-9ced-2e7e3a3461ec-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph4lw\" (UID: \"be2d95d2-5a8b-410e-9ced-2e7e3a3461ec\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph4lw" Oct 30 17:56:09.859145 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:09.859102 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv82k\" (UniqueName: \"kubernetes.io/projected/be2d95d2-5a8b-410e-9ced-2e7e3a3461ec-kube-api-access-fv82k\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph4lw\" (UID: \"be2d95d2-5a8b-410e-9ced-2e7e3a3461ec\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph4lw" Oct 30 17:56:09.911626 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:09.911555 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4dzmb"] Oct 30 17:56:09.922838 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:56:09.922796 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4da73a77_4556_4d95_8e25_5eb2d1c7dc6b.slice/crio-ae7d44d0b18c148be167f3242ade8f457ec7005f9b2a1f1e6df6c5be25f1b77a WatchSource:0}: Error finding container ae7d44d0b18c148be167f3242ade8f457ec7005f9b2a1f1e6df6c5be25f1b77a: Status 404 returned error can't find the container with id ae7d44d0b18c148be167f3242ade8f457ec7005f9b2a1f1e6df6c5be25f1b77a Oct 30 17:56:09.931748 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:09.931567 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph4lw" Oct 30 17:56:10.266939 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:10.266833 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph4lw"] Oct 30 17:56:10.272714 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:56:10.272679 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe2d95d2_5a8b_410e_9ced_2e7e3a3461ec.slice/crio-cb2c4d934b2962b63bd07c7d21e4a90301b954c48ff192a792ccffa6253350d5 WatchSource:0}: Error finding container cb2c4d934b2962b63bd07c7d21e4a90301b954c48ff192a792ccffa6253350d5: Status 404 returned error can't find the container with id cb2c4d934b2962b63bd07c7d21e4a90301b954c48ff192a792ccffa6253350d5 Oct 30 17:56:10.700146 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:10.700013 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph4lw" event={"ID":"be2d95d2-5a8b-410e-9ced-2e7e3a3461ec","Type":"ContainerStarted","Data":"cb2c4d934b2962b63bd07c7d21e4a90301b954c48ff192a792ccffa6253350d5"} Oct 30 17:56:10.702372 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:10.702273 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4dzmb" event={"ID":"4da73a77-4556-4d95-8e25-5eb2d1c7dc6b","Type":"ContainerStarted","Data":"ae7d44d0b18c148be167f3242ade8f457ec7005f9b2a1f1e6df6c5be25f1b77a"} Oct 30 17:56:13.707439 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:13.706610 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 17:56:13.707439 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:13.706706 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 17:56:13.707439 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:13.706752 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 17:56:13.830117 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:13.829432 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 17:56:13.830117 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:13.829527 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 17:56:13.830117 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:13.829571 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 17:56:14.733293 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:14.732438 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" event={"ID":"877f995b-cecd-4842-ba49-963b5c1f0590","Type":"ContainerStarted","Data":"c2e0e766faa984876b8740d02b851d2c57e82d5895e6d6fc47cf7946e6b29243"} Oct 30 17:56:14.736419 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:14.735841 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" event={"ID":"c0d316f7-79bb-4e92-be1e-306552cc08b6","Type":"ContainerStarted","Data":"8e3cc89db8afac75210239ce9725f99727895c2d2d2eaae4ccd1c57fad5b4294"} Oct 30 17:56:14.790126 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:14.790035 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" podStartSLOduration=2.699297377 podStartE2EDuration="7.79001757s" totalImagesPullingTime="5.090720193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:14.78853711 +0000 UTC m=+227.276174146" watchObservedRunningTime="2026-10-30 17:56:14.79001757 +0000 UTC m=+227.277654606" Oct 30 17:56:14.833895 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:14.833280 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" podStartSLOduration=2.471550574 podStartE2EDuration="7.833257069s" totalImagesPullingTime="5.361706495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:14.831500131 +0000 UTC m=+227.319137175" watchObservedRunningTime="2026-10-30 17:56:14.833257069 +0000 UTC m=+227.320894111" Oct 30 17:56:15.140410 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:15.140381 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:56:15.146303 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:15.146279 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:56:15.294189 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:15.294072 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" Oct 30 17:56:15.313347 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:15.313076 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" Oct 30 17:56:15.742851 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:15.742816 4455 generic.go:356] "Generic (PLEG): container finished" podID="820f4cbf-c8c9-40f4-bf53-230143511533" containerID="2981eb3a12fd95077a3d89abcd0f90749b5f47c3aa8f5844c68ae30fdf2f29bb" exitCode=0 Oct 30 17:56:15.743315 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:15.742900 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142cefpdp4n" event={"ID":"820f4cbf-c8c9-40f4-bf53-230143511533","Type":"ContainerDied","Data":"2981eb3a12fd95077a3d89abcd0f90749b5f47c3aa8f5844c68ae30fdf2f29bb"} Oct 30 17:56:15.745231 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:15.745121 4455 generic.go:356] "Generic (PLEG): container finished" podID="b6aa9825-1700-4360-9082-42ba116a2727" containerID="9d6669f05291513a90a87519812bf8dcea10181776b711cb68560dee25860e58" exitCode=0 Oct 30 17:56:15.745231 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:15.745204 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644chl46" event={"ID":"b6aa9825-1700-4360-9082-42ba116a2727","Type":"ContainerDied","Data":"9d6669f05291513a90a87519812bf8dcea10181776b711cb68560dee25860e58"} Oct 30 17:56:15.745376 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:15.745291 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:56:15.745681 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:15.745652 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" Oct 30 17:56:15.746638 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:15.746613 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p" Oct 30 17:56:15.746737 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:15.746724 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:56:16.756870 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:16.756830 4455 generic.go:356] "Generic (PLEG): container finished" podID="820f4cbf-c8c9-40f4-bf53-230143511533" containerID="bc7c3c71719847ab5d7999c39cd25381249fa00c700a14a2378e8a00924e6116" exitCode=0 Oct 30 17:56:16.757376 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:16.756959 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142cefpdp4n" event={"ID":"820f4cbf-c8c9-40f4-bf53-230143511533","Type":"ContainerDied","Data":"bc7c3c71719847ab5d7999c39cd25381249fa00c700a14a2378e8a00924e6116"} Oct 30 17:56:16.759036 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:16.758889 4455 generic.go:356] "Generic (PLEG): container finished" podID="be2d95d2-5a8b-410e-9ced-2e7e3a3461ec" containerID="2dc19ae34f0888376520f56af8d067310a4a51cc2a8c9f9feb6e9ae8d53532c1" exitCode=0 Oct 30 17:56:16.759036 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:16.758957 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph4lw" event={"ID":"be2d95d2-5a8b-410e-9ced-2e7e3a3461ec","Type":"ContainerDied","Data":"2dc19ae34f0888376520f56af8d067310a4a51cc2a8c9f9feb6e9ae8d53532c1"} Oct 30 17:56:16.760746 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:16.760726 4455 generic.go:356] "Generic (PLEG): container finished" podID="4da73a77-4556-4d95-8e25-5eb2d1c7dc6b" containerID="ae2e494bd936811629e80f6eaa3171ebb713caf57a46c87fad83b84508466ba9" exitCode=0 Oct 30 17:56:16.760852 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:16.760820 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4dzmb" event={"ID":"4da73a77-4556-4d95-8e25-5eb2d1c7dc6b","Type":"ContainerDied","Data":"ae2e494bd936811629e80f6eaa3171ebb713caf57a46c87fad83b84508466ba9"} Oct 30 17:56:17.770235 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:17.770197 4455 generic.go:356] "Generic (PLEG): container finished" podID="be2d95d2-5a8b-410e-9ced-2e7e3a3461ec" containerID="814d12d464f019d4ad59718bea7c3b6aadf692424ee087be9a937ed8118100a1" exitCode=0 Oct 30 17:56:17.771789 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:17.771761 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph4lw" event={"ID":"be2d95d2-5a8b-410e-9ced-2e7e3a3461ec","Type":"ContainerDied","Data":"814d12d464f019d4ad59718bea7c3b6aadf692424ee087be9a937ed8118100a1"} Oct 30 17:56:18.779862 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:18.779768 4455 generic.go:356] "Generic (PLEG): container finished" podID="b6aa9825-1700-4360-9082-42ba116a2727" containerID="091b87a190ce56faa698caf6ccd8425f7e8604f7a4cea8863ff4f6b3ca11eb3c" exitCode=0 Oct 30 17:56:18.779862 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:18.779801 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644chl46" event={"ID":"b6aa9825-1700-4360-9082-42ba116a2727","Type":"ContainerDied","Data":"091b87a190ce56faa698caf6ccd8425f7e8604f7a4cea8863ff4f6b3ca11eb3c"} Oct 30 17:56:19.788587 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:19.788546 4455 generic.go:356] "Generic (PLEG): container finished" podID="4da73a77-4556-4d95-8e25-5eb2d1c7dc6b" containerID="730fa9ab10ecea9e9b258a36515c07fb03fa0ac74ac67dab921c7cf98552d5a0" exitCode=0 Oct 30 17:56:19.789070 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:19.788626 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4dzmb" event={"ID":"4da73a77-4556-4d95-8e25-5eb2d1c7dc6b","Type":"ContainerDied","Data":"730fa9ab10ecea9e9b258a36515c07fb03fa0ac74ac67dab921c7cf98552d5a0"} Oct 30 17:56:25.830894 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:25.830360 4455 generic.go:356] "Generic (PLEG): container finished" podID="820f4cbf-c8c9-40f4-bf53-230143511533" containerID="bd7e134324dfab5321fbcdb05b37cdfeddfb608f8f56b9a693e7069629451dbc" exitCode=0 Oct 30 17:56:25.830894 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:25.830515 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142cefpdp4n" event={"ID":"820f4cbf-c8c9-40f4-bf53-230143511533","Type":"ContainerDied","Data":"bd7e134324dfab5321fbcdb05b37cdfeddfb608f8f56b9a693e7069629451dbc"} Oct 30 17:56:26.992810 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:26.992781 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142cefpdp4n" Oct 30 17:56:27.106864 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:27.106822 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/820f4cbf-c8c9-40f4-bf53-230143511533-util\" (UniqueName: \"kubernetes.io/empty-dir/820f4cbf-c8c9-40f4-bf53-230143511533-util\") pod \"820f4cbf-c8c9-40f4-bf53-230143511533\" (UID: \"820f4cbf-c8c9-40f4-bf53-230143511533\") " Oct 30 17:56:27.107064 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:27.106910 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/820f4cbf-c8c9-40f4-bf53-230143511533-kube-api-access-kjrtz\" (UniqueName: \"kubernetes.io/projected/820f4cbf-c8c9-40f4-bf53-230143511533-kube-api-access-kjrtz\") pod \"820f4cbf-c8c9-40f4-bf53-230143511533\" (UID: \"820f4cbf-c8c9-40f4-bf53-230143511533\") " Oct 30 17:56:27.107064 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:27.106993 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/820f4cbf-c8c9-40f4-bf53-230143511533-bundle\" (UniqueName: \"kubernetes.io/empty-dir/820f4cbf-c8c9-40f4-bf53-230143511533-bundle\") pod \"820f4cbf-c8c9-40f4-bf53-230143511533\" (UID: \"820f4cbf-c8c9-40f4-bf53-230143511533\") " Oct 30 17:56:27.107694 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:27.107669 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820f4cbf-c8c9-40f4-bf53-230143511533-bundle" pod "820f4cbf-c8c9-40f4-bf53-230143511533" (UID: "820f4cbf-c8c9-40f4-bf53-230143511533"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:56:27.109700 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:27.109659 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/820f4cbf-c8c9-40f4-bf53-230143511533-kube-api-access-kjrtz" pod "820f4cbf-c8c9-40f4-bf53-230143511533" (UID: "820f4cbf-c8c9-40f4-bf53-230143511533"). InnerVolumeSpecName "kube-api-access-kjrtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:56:27.111521 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:27.111492 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820f4cbf-c8c9-40f4-bf53-230143511533-util" pod "820f4cbf-c8c9-40f4-bf53-230143511533" (UID: "820f4cbf-c8c9-40f4-bf53-230143511533"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:56:27.208413 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:27.208364 4455 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/820f4cbf-c8c9-40f4-bf53-230143511533-bundle\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:56:27.208413 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:27.208401 4455 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/820f4cbf-c8c9-40f4-bf53-230143511533-util\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:56:27.208657 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:27.208429 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjrtz\" (UniqueName: \"kubernetes.io/projected/820f4cbf-c8c9-40f4-bf53-230143511533-kube-api-access-kjrtz\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:56:27.844023 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:27.843939 4455 generic.go:356] "Generic (PLEG): container finished" podID="b6aa9825-1700-4360-9082-42ba116a2727" containerID="4c575607187e6ad2e1651676a4d77277712a85ad5198471b971899cf5aac2ca4" exitCode=0 Oct 30 17:56:27.844192 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:27.844024 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644chl46" event={"ID":"b6aa9825-1700-4360-9082-42ba116a2727","Type":"ContainerDied","Data":"4c575607187e6ad2e1651676a4d77277712a85ad5198471b971899cf5aac2ca4"} Oct 30 17:56:27.846602 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:27.846573 4455 generic.go:356] "Generic (PLEG): container finished" podID="be2d95d2-5a8b-410e-9ced-2e7e3a3461ec" containerID="9b85bcb27e7d8a05849e156037e9be39e05985b173cb24c2340b044745d01f53" exitCode=0 Oct 30 17:56:27.846741 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:27.846657 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph4lw" event={"ID":"be2d95d2-5a8b-410e-9ced-2e7e3a3461ec","Type":"ContainerDied","Data":"9b85bcb27e7d8a05849e156037e9be39e05985b173cb24c2340b044745d01f53"} Oct 30 17:56:27.848666 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:27.848642 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142cefpdp4n" event={"ID":"820f4cbf-c8c9-40f4-bf53-230143511533","Type":"ContainerDied","Data":"fc77e4701814c43a9c39396a8edef9d6e99082b67360494a2b00f9c526955a54"} Oct 30 17:56:27.848791 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:27.848684 4455 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc77e4701814c43a9c39396a8edef9d6e99082b67360494a2b00f9c526955a54" Oct 30 17:56:27.848791 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:27.848649 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142cefpdp4n" Oct 30 17:56:28.855191 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:28.855154 4455 generic.go:356] "Generic (PLEG): container finished" podID="4da73a77-4556-4d95-8e25-5eb2d1c7dc6b" containerID="16bf4f67f0f17e2f9704e51001bff2333ff0b0b17f3fc87459033e45d2ad07aa" exitCode=0 Oct 30 17:56:28.855662 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:28.855246 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4dzmb" event={"ID":"4da73a77-4556-4d95-8e25-5eb2d1c7dc6b","Type":"ContainerDied","Data":"16bf4f67f0f17e2f9704e51001bff2333ff0b0b17f3fc87459033e45d2ad07aa"} Oct 30 17:56:29.009127 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:29.009106 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644chl46" Oct 30 17:56:29.013057 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:29.013039 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph4lw" Oct 30 17:56:29.128817 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:29.128739 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be2d95d2-5a8b-410e-9ced-2e7e3a3461ec-util\" (UniqueName: \"kubernetes.io/empty-dir/be2d95d2-5a8b-410e-9ced-2e7e3a3461ec-util\") pod \"be2d95d2-5a8b-410e-9ced-2e7e3a3461ec\" (UID: \"be2d95d2-5a8b-410e-9ced-2e7e3a3461ec\") " Oct 30 17:56:29.128817 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:29.128785 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be2d95d2-5a8b-410e-9ced-2e7e3a3461ec-kube-api-access-fv82k\" (UniqueName: \"kubernetes.io/projected/be2d95d2-5a8b-410e-9ced-2e7e3a3461ec-kube-api-access-fv82k\") pod \"be2d95d2-5a8b-410e-9ced-2e7e3a3461ec\" (UID: \"be2d95d2-5a8b-410e-9ced-2e7e3a3461ec\") " Oct 30 17:56:29.128817 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:29.128809 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6aa9825-1700-4360-9082-42ba116a2727-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6aa9825-1700-4360-9082-42ba116a2727-bundle\") pod \"b6aa9825-1700-4360-9082-42ba116a2727\" (UID: \"b6aa9825-1700-4360-9082-42ba116a2727\") " Oct 30 17:56:29.129101 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:29.128827 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6aa9825-1700-4360-9082-42ba116a2727-util\" (UniqueName: \"kubernetes.io/empty-dir/b6aa9825-1700-4360-9082-42ba116a2727-util\") pod \"b6aa9825-1700-4360-9082-42ba116a2727\" (UID: \"b6aa9825-1700-4360-9082-42ba116a2727\") " Oct 30 17:56:29.129101 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:29.128886 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be2d95d2-5a8b-410e-9ced-2e7e3a3461ec-bundle\" (UniqueName: \"kubernetes.io/empty-dir/be2d95d2-5a8b-410e-9ced-2e7e3a3461ec-bundle\") pod \"be2d95d2-5a8b-410e-9ced-2e7e3a3461ec\" (UID: \"be2d95d2-5a8b-410e-9ced-2e7e3a3461ec\") " Oct 30 17:56:29.129101 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:29.128912 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6aa9825-1700-4360-9082-42ba116a2727-kube-api-access-pldts\" (UniqueName: \"kubernetes.io/projected/b6aa9825-1700-4360-9082-42ba116a2727-kube-api-access-pldts\") pod \"b6aa9825-1700-4360-9082-42ba116a2727\" (UID: \"b6aa9825-1700-4360-9082-42ba116a2727\") " Oct 30 17:56:29.129287 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:29.129252 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6aa9825-1700-4360-9082-42ba116a2727-bundle" pod "b6aa9825-1700-4360-9082-42ba116a2727" (UID: "b6aa9825-1700-4360-9082-42ba116a2727"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:56:29.129450 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:29.129428 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be2d95d2-5a8b-410e-9ced-2e7e3a3461ec-bundle" pod "be2d95d2-5a8b-410e-9ced-2e7e3a3461ec" (UID: "be2d95d2-5a8b-410e-9ced-2e7e3a3461ec"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:56:29.131074 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:29.131042 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be2d95d2-5a8b-410e-9ced-2e7e3a3461ec-kube-api-access-fv82k" pod "be2d95d2-5a8b-410e-9ced-2e7e3a3461ec" (UID: "be2d95d2-5a8b-410e-9ced-2e7e3a3461ec"). InnerVolumeSpecName "kube-api-access-fv82k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:56:29.131550 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:29.131525 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6aa9825-1700-4360-9082-42ba116a2727-kube-api-access-pldts" pod "b6aa9825-1700-4360-9082-42ba116a2727" (UID: "b6aa9825-1700-4360-9082-42ba116a2727"). InnerVolumeSpecName "kube-api-access-pldts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:56:29.134771 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:29.134747 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6aa9825-1700-4360-9082-42ba116a2727-util" pod "b6aa9825-1700-4360-9082-42ba116a2727" (UID: "b6aa9825-1700-4360-9082-42ba116a2727"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:56:29.135968 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:29.135949 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be2d95d2-5a8b-410e-9ced-2e7e3a3461ec-util" pod "be2d95d2-5a8b-410e-9ced-2e7e3a3461ec" (UID: "be2d95d2-5a8b-410e-9ced-2e7e3a3461ec"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:56:29.229800 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:29.229757 4455 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be2d95d2-5a8b-410e-9ced-2e7e3a3461ec-util\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:56:29.229800 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:29.229793 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv82k\" (UniqueName: \"kubernetes.io/projected/be2d95d2-5a8b-410e-9ced-2e7e3a3461ec-kube-api-access-fv82k\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:56:29.229800 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:29.229808 4455 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6aa9825-1700-4360-9082-42ba116a2727-bundle\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:56:29.230038 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:29.229817 4455 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6aa9825-1700-4360-9082-42ba116a2727-util\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:56:29.230038 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:29.229825 4455 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be2d95d2-5a8b-410e-9ced-2e7e3a3461ec-bundle\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:56:29.230038 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:29.229834 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pldts\" (UniqueName: \"kubernetes.io/projected/b6aa9825-1700-4360-9082-42ba116a2727-kube-api-access-pldts\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:56:29.861447 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:29.861408 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644chl46" event={"ID":"b6aa9825-1700-4360-9082-42ba116a2727","Type":"ContainerDied","Data":"6dfb83ad57e98607d04dcec3b10555af1934a6b5c12d758fd83e1022ef4591c8"} Oct 30 17:56:29.861447 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:29.861445 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644chl46" Oct 30 17:56:29.861938 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:29.861466 4455 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dfb83ad57e98607d04dcec3b10555af1934a6b5c12d758fd83e1022ef4591c8" Oct 30 17:56:29.863813 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:29.863520 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph4lw" Oct 30 17:56:29.863813 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:29.863562 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ph4lw" event={"ID":"be2d95d2-5a8b-410e-9ced-2e7e3a3461ec","Type":"ContainerDied","Data":"cb2c4d934b2962b63bd07c7d21e4a90301b954c48ff192a792ccffa6253350d5"} Oct 30 17:56:29.863813 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:29.863594 4455 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb2c4d934b2962b63bd07c7d21e4a90301b954c48ff192a792ccffa6253350d5" Oct 30 17:56:29.980147 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:29.980120 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4dzmb" Oct 30 17:56:30.036155 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:30.036119 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4da73a77-4556-4d95-8e25-5eb2d1c7dc6b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4da73a77-4556-4d95-8e25-5eb2d1c7dc6b-bundle\") pod \"4da73a77-4556-4d95-8e25-5eb2d1c7dc6b\" (UID: \"4da73a77-4556-4d95-8e25-5eb2d1c7dc6b\") " Oct 30 17:56:30.036354 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:30.036311 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4da73a77-4556-4d95-8e25-5eb2d1c7dc6b-util\" (UniqueName: \"kubernetes.io/empty-dir/4da73a77-4556-4d95-8e25-5eb2d1c7dc6b-util\") pod \"4da73a77-4556-4d95-8e25-5eb2d1c7dc6b\" (UID: \"4da73a77-4556-4d95-8e25-5eb2d1c7dc6b\") " Oct 30 17:56:30.036429 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:30.036379 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4da73a77-4556-4d95-8e25-5eb2d1c7dc6b-kube-api-access-92lh9\" (UniqueName: \"kubernetes.io/projected/4da73a77-4556-4d95-8e25-5eb2d1c7dc6b-kube-api-access-92lh9\") pod \"4da73a77-4556-4d95-8e25-5eb2d1c7dc6b\" (UID: \"4da73a77-4556-4d95-8e25-5eb2d1c7dc6b\") " Oct 30 17:56:30.036908 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:30.036873 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4da73a77-4556-4d95-8e25-5eb2d1c7dc6b-bundle" pod "4da73a77-4556-4d95-8e25-5eb2d1c7dc6b" (UID: "4da73a77-4556-4d95-8e25-5eb2d1c7dc6b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:56:30.038649 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:30.038621 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4da73a77-4556-4d95-8e25-5eb2d1c7dc6b-kube-api-access-92lh9" pod "4da73a77-4556-4d95-8e25-5eb2d1c7dc6b" (UID: "4da73a77-4556-4d95-8e25-5eb2d1c7dc6b"). InnerVolumeSpecName "kube-api-access-92lh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:56:30.040174 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:30.040151 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4da73a77-4556-4d95-8e25-5eb2d1c7dc6b-util" pod "4da73a77-4556-4d95-8e25-5eb2d1c7dc6b" (UID: "4da73a77-4556-4d95-8e25-5eb2d1c7dc6b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:56:30.137882 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:30.137846 4455 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4da73a77-4556-4d95-8e25-5eb2d1c7dc6b-util\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:56:30.137882 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:30.137875 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92lh9\" (UniqueName: \"kubernetes.io/projected/4da73a77-4556-4d95-8e25-5eb2d1c7dc6b-kube-api-access-92lh9\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:56:30.137882 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:30.137888 4455 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4da73a77-4556-4d95-8e25-5eb2d1c7dc6b-bundle\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:56:30.383600 ip-10-0-130-84 kubenswrapper[4455]: E1030 17:56:30.383569 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 17:56:30.870269 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:30.870230 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4dzmb" event={"ID":"4da73a77-4556-4d95-8e25-5eb2d1c7dc6b","Type":"ContainerDied","Data":"ae7d44d0b18c148be167f3242ade8f457ec7005f9b2a1f1e6df6c5be25f1b77a"} Oct 30 17:56:30.870789 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:30.870286 4455 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae7d44d0b18c148be167f3242ade8f457ec7005f9b2a1f1e6df6c5be25f1b77a" Oct 30 17:56:30.870789 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:30.870247 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4dzmb" Oct 30 17:56:31.320356 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:31.320290 4455 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-84.ec2.internal" podUID="da45dfd37aee5c345465266021945ad3" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Oct 30 17:56:32.196824 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.196785 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-44nvn"] Oct 30 17:56:32.197378 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.197359 4455 cpu_manager.go:414] "containerMap: removing container" podUID="be2d95d2-5a8b-410e-9ced-2e7e3a3461ec" containerName="extract" Oct 30 17:56:32.197467 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.197380 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be2d95d2-5a8b-410e-9ced-2e7e3a3461ec" containerName="extract" Oct 30 17:56:32.197467 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.197397 4455 cpu_manager.go:414] "containerMap: removing container" podUID="4da73a77-4556-4d95-8e25-5eb2d1c7dc6b" containerName="extract" Oct 30 17:56:32.197467 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.197404 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4da73a77-4556-4d95-8e25-5eb2d1c7dc6b" containerName="extract" Oct 30 17:56:32.197467 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.197427 4455 cpu_manager.go:414] "containerMap: removing container" podUID="820f4cbf-c8c9-40f4-bf53-230143511533" containerName="util" Oct 30 17:56:32.197467 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.197434 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="820f4cbf-c8c9-40f4-bf53-230143511533" containerName="util" Oct 30 17:56:32.197467 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.197446 4455 cpu_manager.go:414] "containerMap: removing container" podUID="b6aa9825-1700-4360-9082-42ba116a2727" containerName="util" Oct 30 17:56:32.197467 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.197453 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6aa9825-1700-4360-9082-42ba116a2727" containerName="util" Oct 30 17:56:32.197467 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.197461 4455 cpu_manager.go:414] "containerMap: removing container" podUID="be2d95d2-5a8b-410e-9ced-2e7e3a3461ec" containerName="util" Oct 30 17:56:32.197467 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.197468 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be2d95d2-5a8b-410e-9ced-2e7e3a3461ec" containerName="util" Oct 30 17:56:32.197893 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.197482 4455 cpu_manager.go:414] "containerMap: removing container" podUID="b6aa9825-1700-4360-9082-42ba116a2727" containerName="extract" Oct 30 17:56:32.197893 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.197489 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6aa9825-1700-4360-9082-42ba116a2727" containerName="extract" Oct 30 17:56:32.197893 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.197502 4455 cpu_manager.go:414] "containerMap: removing container" podUID="4da73a77-4556-4d95-8e25-5eb2d1c7dc6b" containerName="pull" Oct 30 17:56:32.197893 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.197510 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4da73a77-4556-4d95-8e25-5eb2d1c7dc6b" containerName="pull" Oct 30 17:56:32.197893 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.197525 4455 cpu_manager.go:414] "containerMap: removing container" podUID="820f4cbf-c8c9-40f4-bf53-230143511533" containerName="pull" Oct 30 17:56:32.197893 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.197532 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="820f4cbf-c8c9-40f4-bf53-230143511533" containerName="pull" Oct 30 17:56:32.197893 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.197551 4455 cpu_manager.go:414] "containerMap: removing container" podUID="4da73a77-4556-4d95-8e25-5eb2d1c7dc6b" containerName="util" Oct 30 17:56:32.197893 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.197558 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4da73a77-4556-4d95-8e25-5eb2d1c7dc6b" containerName="util" Oct 30 17:56:32.197893 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.197573 4455 cpu_manager.go:414] "containerMap: removing container" podUID="be2d95d2-5a8b-410e-9ced-2e7e3a3461ec" containerName="pull" Oct 30 17:56:32.197893 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.197580 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be2d95d2-5a8b-410e-9ced-2e7e3a3461ec" containerName="pull" Oct 30 17:56:32.197893 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.197589 4455 cpu_manager.go:414] "containerMap: removing container" podUID="b6aa9825-1700-4360-9082-42ba116a2727" containerName="pull" Oct 30 17:56:32.197893 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.197595 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6aa9825-1700-4360-9082-42ba116a2727" containerName="pull" Oct 30 17:56:32.197893 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.197605 4455 cpu_manager.go:414] "containerMap: removing container" podUID="820f4cbf-c8c9-40f4-bf53-230143511533" containerName="extract" Oct 30 17:56:32.197893 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.197611 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="820f4cbf-c8c9-40f4-bf53-230143511533" containerName="extract" Oct 30 17:56:32.197893 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.197700 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="820f4cbf-c8c9-40f4-bf53-230143511533" containerName="extract" Oct 30 17:56:32.197893 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.197712 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="be2d95d2-5a8b-410e-9ced-2e7e3a3461ec" containerName="extract" Oct 30 17:56:32.197893 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.197722 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="4da73a77-4556-4d95-8e25-5eb2d1c7dc6b" containerName="extract" Oct 30 17:56:32.197893 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.197734 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6aa9825-1700-4360-9082-42ba116a2727" containerName="extract" Oct 30 17:56:32.205146 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.205124 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-44nvn" Oct 30 17:56:32.219365 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.219339 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 17:56:32.219481 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.219388 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 17:56:32.230594 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.230567 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-v2gs7\"" Oct 30 17:56:32.230594 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.230578 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 17:56:32.230727 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.230643 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 17:56:32.230727 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.230643 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 17:56:32.258144 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.258099 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7gsc\" (UniqueName: \"kubernetes.io/projected/99902e75-ed4c-4ac2-afa7-c00d963ab113-kube-api-access-s7gsc\") pod \"opendatahub-operator-controller-manager-64bcb8d955-44nvn\" (UID: \"99902e75-ed4c-4ac2-afa7-c00d963ab113\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-44nvn" Oct 30 17:56:32.258350 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.258194 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/99902e75-ed4c-4ac2-afa7-c00d963ab113-manifests\") pod \"opendatahub-operator-controller-manager-64bcb8d955-44nvn\" (UID: \"99902e75-ed4c-4ac2-afa7-c00d963ab113\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-44nvn" Oct 30 17:56:32.258350 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.258235 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/99902e75-ed4c-4ac2-afa7-c00d963ab113-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64bcb8d955-44nvn\" (UID: \"99902e75-ed4c-4ac2-afa7-c00d963ab113\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-44nvn" Oct 30 17:56:32.258350 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.258302 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99902e75-ed4c-4ac2-afa7-c00d963ab113-tmp\") pod \"opendatahub-operator-controller-manager-64bcb8d955-44nvn\" (UID: \"99902e75-ed4c-4ac2-afa7-c00d963ab113\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-44nvn" Oct 30 17:56:32.258606 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.258366 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/99902e75-ed4c-4ac2-afa7-c00d963ab113-webhook-cert\") pod \"opendatahub-operator-controller-manager-64bcb8d955-44nvn\" (UID: \"99902e75-ed4c-4ac2-afa7-c00d963ab113\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-44nvn" Oct 30 17:56:32.258606 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.258390 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/99902e75-ed4c-4ac2-afa7-c00d963ab113-metrics-cert\") pod \"opendatahub-operator-controller-manager-64bcb8d955-44nvn\" (UID: \"99902e75-ed4c-4ac2-afa7-c00d963ab113\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-44nvn" Oct 30 17:56:32.258711 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.258678 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-44nvn"] Oct 30 17:56:32.359854 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.359821 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/99902e75-ed4c-4ac2-afa7-c00d963ab113-manifests\") pod \"opendatahub-operator-controller-manager-64bcb8d955-44nvn\" (UID: \"99902e75-ed4c-4ac2-afa7-c00d963ab113\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-44nvn" Oct 30 17:56:32.359854 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.359860 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/99902e75-ed4c-4ac2-afa7-c00d963ab113-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64bcb8d955-44nvn\" (UID: \"99902e75-ed4c-4ac2-afa7-c00d963ab113\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-44nvn" Oct 30 17:56:32.360100 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.359901 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99902e75-ed4c-4ac2-afa7-c00d963ab113-tmp\") pod \"opendatahub-operator-controller-manager-64bcb8d955-44nvn\" (UID: \"99902e75-ed4c-4ac2-afa7-c00d963ab113\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-44nvn" Oct 30 17:56:32.360100 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.359929 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/99902e75-ed4c-4ac2-afa7-c00d963ab113-webhook-cert\") pod \"opendatahub-operator-controller-manager-64bcb8d955-44nvn\" (UID: \"99902e75-ed4c-4ac2-afa7-c00d963ab113\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-44nvn" Oct 30 17:56:32.360100 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.359944 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/99902e75-ed4c-4ac2-afa7-c00d963ab113-metrics-cert\") pod \"opendatahub-operator-controller-manager-64bcb8d955-44nvn\" (UID: \"99902e75-ed4c-4ac2-afa7-c00d963ab113\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-44nvn" Oct 30 17:56:32.360233 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.360109 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7gsc\" (UniqueName: \"kubernetes.io/projected/99902e75-ed4c-4ac2-afa7-c00d963ab113-kube-api-access-s7gsc\") pod \"opendatahub-operator-controller-manager-64bcb8d955-44nvn\" (UID: \"99902e75-ed4c-4ac2-afa7-c00d963ab113\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-44nvn" Oct 30 17:56:32.360372 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.360274 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/99902e75-ed4c-4ac2-afa7-c00d963ab113-manifests\") pod \"opendatahub-operator-controller-manager-64bcb8d955-44nvn\" (UID: \"99902e75-ed4c-4ac2-afa7-c00d963ab113\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-44nvn" Oct 30 17:56:32.362625 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.362593 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99902e75-ed4c-4ac2-afa7-c00d963ab113-tmp\") pod \"opendatahub-operator-controller-manager-64bcb8d955-44nvn\" (UID: \"99902e75-ed4c-4ac2-afa7-c00d963ab113\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-44nvn" Oct 30 17:56:32.362807 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.362789 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/99902e75-ed4c-4ac2-afa7-c00d963ab113-webhook-cert\") pod \"opendatahub-operator-controller-manager-64bcb8d955-44nvn\" (UID: \"99902e75-ed4c-4ac2-afa7-c00d963ab113\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-44nvn" Oct 30 17:56:32.362851 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.362791 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/99902e75-ed4c-4ac2-afa7-c00d963ab113-metrics-cert\") pod \"opendatahub-operator-controller-manager-64bcb8d955-44nvn\" (UID: \"99902e75-ed4c-4ac2-afa7-c00d963ab113\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-44nvn" Oct 30 17:56:32.362851 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.362791 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/99902e75-ed4c-4ac2-afa7-c00d963ab113-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64bcb8d955-44nvn\" (UID: \"99902e75-ed4c-4ac2-afa7-c00d963ab113\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-44nvn" Oct 30 17:56:32.386871 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.386845 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7gsc\" (UniqueName: \"kubernetes.io/projected/99902e75-ed4c-4ac2-afa7-c00d963ab113-kube-api-access-s7gsc\") pod \"opendatahub-operator-controller-manager-64bcb8d955-44nvn\" (UID: \"99902e75-ed4c-4ac2-afa7-c00d963ab113\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-44nvn" Oct 30 17:56:32.527461 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.527431 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-44nvn" Oct 30 17:56:32.883809 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:32.883782 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-44nvn"] Oct 30 17:56:32.886666 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:56:32.886626 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99902e75_ed4c_4ac2_afa7_c00d963ab113.slice/crio-69c1a351017f2d00a67a2ce21e2c3a6cd061be3858d7ec20d91a16c3a0adb84c WatchSource:0}: Error finding container 69c1a351017f2d00a67a2ce21e2c3a6cd061be3858d7ec20d91a16c3a0adb84c: Status 404 returned error can't find the container with id 69c1a351017f2d00a67a2ce21e2c3a6cd061be3858d7ec20d91a16c3a0adb84c Oct 30 17:56:33.887590 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:33.887546 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-44nvn" event={"ID":"99902e75-ed4c-4ac2-afa7-c00d963ab113","Type":"ContainerStarted","Data":"69c1a351017f2d00a67a2ce21e2c3a6cd061be3858d7ec20d91a16c3a0adb84c"} Oct 30 17:56:36.904035 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:36.903999 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-44nvn" event={"ID":"99902e75-ed4c-4ac2-afa7-c00d963ab113","Type":"ContainerStarted","Data":"a14db793968cf2636779c042048a3bb95af773d5194a22d76e88f368596743c9"} Oct 30 17:56:37.910447 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:37.910413 4455 generic.go:356] "Generic (PLEG): container finished" podID="99902e75-ed4c-4ac2-afa7-c00d963ab113" containerID="a14db793968cf2636779c042048a3bb95af773d5194a22d76e88f368596743c9" exitCode=0 Oct 30 17:56:37.910888 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:37.910499 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-44nvn" event={"ID":"99902e75-ed4c-4ac2-afa7-c00d963ab113","Type":"ContainerDied","Data":"a14db793968cf2636779c042048a3bb95af773d5194a22d76e88f368596743c9"} Oct 30 17:56:38.917292 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:38.917259 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-44nvn" event={"ID":"99902e75-ed4c-4ac2-afa7-c00d963ab113","Type":"ContainerStarted","Data":"3dc15b9ecaaad590d2e3c550b8a9b61e1be9b7187f6c5187cf44a3aecb2962ce"} Oct 30 17:56:38.917684 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:38.917389 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-44nvn" Oct 30 17:56:39.003715 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:39.003661 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-44nvn" podStartSLOduration=2.5891857910000002 podStartE2EDuration="7.003648688s" totalImagesPullingTime="3.888460945s" totalInitContainerRuntime="526.001952ms" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:38.992055763 +0000 UTC m=+251.479692796" watchObservedRunningTime="2026-10-30 17:56:39.003648688 +0000 UTC m=+251.491285721" Oct 30 17:56:41.315300 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:41.315261 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-89fs5"] Oct 30 17:56:41.317844 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:41.317820 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-89fs5" Oct 30 17:56:41.331703 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:41.331683 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 17:56:41.343098 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:41.343075 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:56:41.343213 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:41.343094 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-bpwh6\"" Oct 30 17:56:41.347510 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:41.347488 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-89fs5"] Oct 30 17:56:41.443032 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:41.442996 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c069815-73a5-4c20-be44-31b7708db07b-tmp\") pod \"jobset-operator-674cd59dc5-89fs5\" (UID: \"9c069815-73a5-4c20-be44-31b7708db07b\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-89fs5" Oct 30 17:56:41.443032 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:41.443030 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t86n\" (UniqueName: \"kubernetes.io/projected/9c069815-73a5-4c20-be44-31b7708db07b-kube-api-access-8t86n\") pod \"jobset-operator-674cd59dc5-89fs5\" (UID: \"9c069815-73a5-4c20-be44-31b7708db07b\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-89fs5" Oct 30 17:56:41.544142 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:41.544110 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c069815-73a5-4c20-be44-31b7708db07b-tmp\") pod \"jobset-operator-674cd59dc5-89fs5\" (UID: \"9c069815-73a5-4c20-be44-31b7708db07b\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-89fs5" Oct 30 17:56:41.544142 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:41.544142 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t86n\" (UniqueName: \"kubernetes.io/projected/9c069815-73a5-4c20-be44-31b7708db07b-kube-api-access-8t86n\") pod \"jobset-operator-674cd59dc5-89fs5\" (UID: \"9c069815-73a5-4c20-be44-31b7708db07b\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-89fs5" Oct 30 17:56:41.544575 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:41.544553 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c069815-73a5-4c20-be44-31b7708db07b-tmp\") pod \"jobset-operator-674cd59dc5-89fs5\" (UID: \"9c069815-73a5-4c20-be44-31b7708db07b\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-89fs5" Oct 30 17:56:41.566488 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:41.566426 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t86n\" (UniqueName: \"kubernetes.io/projected/9c069815-73a5-4c20-be44-31b7708db07b-kube-api-access-8t86n\") pod \"jobset-operator-674cd59dc5-89fs5\" (UID: \"9c069815-73a5-4c20-be44-31b7708db07b\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-89fs5" Oct 30 17:56:41.644175 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:41.644146 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-89fs5" Oct 30 17:56:41.703007 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:41.702980 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76d5f9c5c-fb457"] Oct 30 17:56:41.706294 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:41.706269 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76d5f9c5c-fb457" Oct 30 17:56:41.720599 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:41.720570 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 17:56:41.720599 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:41.720592 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 17:56:41.720787 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:41.720590 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-27zl4\"" Oct 30 17:56:41.733632 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:41.733599 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76d5f9c5c-fb457"] Oct 30 17:56:41.847575 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:41.847486 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwkch\" (UniqueName: \"kubernetes.io/projected/a896d805-773f-4bb0-a8ab-36dfec29af5c-kube-api-access-cwkch\") pod \"kuadrant-operator-controller-manager-76d5f9c5c-fb457\" (UID: \"a896d805-773f-4bb0-a8ab-36dfec29af5c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76d5f9c5c-fb457" Oct 30 17:56:41.941617 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:41.941409 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-89fs5"] Oct 30 17:56:41.944240 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:56:41.944214 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c069815_73a5_4c20_be44_31b7708db07b.slice/crio-04b1a7d82b307ca5ce8ed9a8bfbdc4e78591988477534aaf3924f3db84dcf1e4 WatchSource:0}: Error finding container 04b1a7d82b307ca5ce8ed9a8bfbdc4e78591988477534aaf3924f3db84dcf1e4: Status 404 returned error can't find the container with id 04b1a7d82b307ca5ce8ed9a8bfbdc4e78591988477534aaf3924f3db84dcf1e4 Oct 30 17:56:41.947972 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:41.947942 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwkch\" (UniqueName: \"kubernetes.io/projected/a896d805-773f-4bb0-a8ab-36dfec29af5c-kube-api-access-cwkch\") pod \"kuadrant-operator-controller-manager-76d5f9c5c-fb457\" (UID: \"a896d805-773f-4bb0-a8ab-36dfec29af5c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76d5f9c5c-fb457" Oct 30 17:56:41.970159 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:41.970128 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwkch\" (UniqueName: \"kubernetes.io/projected/a896d805-773f-4bb0-a8ab-36dfec29af5c-kube-api-access-cwkch\") pod \"kuadrant-operator-controller-manager-76d5f9c5c-fb457\" (UID: \"a896d805-773f-4bb0-a8ab-36dfec29af5c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76d5f9c5c-fb457" Oct 30 17:56:42.022125 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:42.022102 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76d5f9c5c-fb457" Oct 30 17:56:42.328522 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:42.328440 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76d5f9c5c-fb457"] Oct 30 17:56:42.331145 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:56:42.331100 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda896d805_773f_4bb0_a8ab_36dfec29af5c.slice/crio-38a26c93d08299a355b30bb4a63f5783a9b34a4b8161be5c510ea0737ac5dfd0 WatchSource:0}: Error finding container 38a26c93d08299a355b30bb4a63f5783a9b34a4b8161be5c510ea0737ac5dfd0: Status 404 returned error can't find the container with id 38a26c93d08299a355b30bb4a63f5783a9b34a4b8161be5c510ea0737ac5dfd0 Oct 30 17:56:42.937294 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:42.937250 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76d5f9c5c-fb457" event={"ID":"a896d805-773f-4bb0-a8ab-36dfec29af5c","Type":"ContainerStarted","Data":"38a26c93d08299a355b30bb4a63f5783a9b34a4b8161be5c510ea0737ac5dfd0"} Oct 30 17:56:42.938792 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:42.938767 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-89fs5" event={"ID":"9c069815-73a5-4c20-be44-31b7708db07b","Type":"ContainerStarted","Data":"04b1a7d82b307ca5ce8ed9a8bfbdc4e78591988477534aaf3924f3db84dcf1e4"} Oct 30 17:56:44.952619 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:44.952584 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-89fs5" event={"ID":"9c069815-73a5-4c20-be44-31b7708db07b","Type":"ContainerStarted","Data":"f9f91729fd43b121d7d46214b9fb3ca91f6996eecfd44afe7fd618744cdb2352"} Oct 30 17:56:44.993592 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:44.993543 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-89fs5" podStartSLOduration=1.596856549 podStartE2EDuration="3.993529737s" totalImagesPullingTime="2.396673188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:44.992505627 +0000 UTC m=+257.480142661" watchObservedRunningTime="2026-10-30 17:56:44.993529737 +0000 UTC m=+257.481166771" Oct 30 17:56:47.968102 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:47.968070 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76d5f9c5c-fb457" event={"ID":"a896d805-773f-4bb0-a8ab-36dfec29af5c","Type":"ContainerStarted","Data":"bd9aba662a0a97d94763267ed6e67a09284d902ad926ca2bc591df19b1c87644"} Oct 30 17:56:47.968589 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:47.968197 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76d5f9c5c-fb457" Oct 30 17:56:48.077034 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:48.076970 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76d5f9c5c-fb457" podStartSLOduration=1.881383416 podStartE2EDuration="7.076955499s" totalImagesPullingTime="5.195572083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:48.065170895 +0000 UTC m=+260.552807926" watchObservedRunningTime="2026-10-30 17:56:48.076955499 +0000 UTC m=+260.564592533" Oct 30 17:56:49.924283 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:49.924247 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-44nvn" Oct 30 17:56:57.783918 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:57.783882 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-57f5c66455-l7hkg"] Oct 30 17:56:57.786791 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:57.786771 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-57f5c66455-l7hkg" Oct 30 17:56:57.795502 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:57.795481 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szfjw\" (UniqueName: \"kubernetes.io/projected/64c72761-8ae7-47a1-aa47-d10cadeb3e34-kube-api-access-szfjw\") pod \"dns-operator-controller-manager-57f5c66455-l7hkg\" (UID: \"64c72761-8ae7-47a1-aa47-d10cadeb3e34\") " pod="kuadrant-system/dns-operator-controller-manager-57f5c66455-l7hkg" Oct 30 17:56:57.809128 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:57.809088 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-57f5c66455-l7hkg"] Oct 30 17:56:57.835277 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:57.835248 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-z8g6g\"" Oct 30 17:56:57.857447 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:57.857422 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 17:56:57.896693 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:57.896669 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szfjw\" (UniqueName: \"kubernetes.io/projected/64c72761-8ae7-47a1-aa47-d10cadeb3e34-kube-api-access-szfjw\") pod \"dns-operator-controller-manager-57f5c66455-l7hkg\" (UID: \"64c72761-8ae7-47a1-aa47-d10cadeb3e34\") " pod="kuadrant-system/dns-operator-controller-manager-57f5c66455-l7hkg" Oct 30 17:56:57.919117 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:57.919092 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szfjw\" (UniqueName: \"kubernetes.io/projected/64c72761-8ae7-47a1-aa47-d10cadeb3e34-kube-api-access-szfjw\") pod \"dns-operator-controller-manager-57f5c66455-l7hkg\" (UID: \"64c72761-8ae7-47a1-aa47-d10cadeb3e34\") " pod="kuadrant-system/dns-operator-controller-manager-57f5c66455-l7hkg" Oct 30 17:56:58.100039 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:58.099948 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-57f5c66455-l7hkg" Oct 30 17:56:58.406579 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:58.406507 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-57f5c66455-l7hkg"] Oct 30 17:56:58.411593 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:56:58.411560 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64c72761_8ae7_47a1_aa47_d10cadeb3e34.slice/crio-031530840e04adce4b543234c2a78e739f5643dcc9ebceb3de03381970c76089 WatchSource:0}: Error finding container 031530840e04adce4b543234c2a78e739f5643dcc9ebceb3de03381970c76089: Status 404 returned error can't find the container with id 031530840e04adce4b543234c2a78e739f5643dcc9ebceb3de03381970c76089 Oct 30 17:56:58.790684 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:58.790636 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-779bff956b-bmwkw"] Oct 30 17:56:58.794359 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:58.794337 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-779bff956b-bmwkw" Oct 30 17:56:58.803466 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:58.803443 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h284\" (UniqueName: \"kubernetes.io/projected/8e4d9f9b-bd72-4d8b-a533-79334334789f-kube-api-access-8h284\") pod \"authorino-operator-779bff956b-bmwkw\" (UID: \"8e4d9f9b-bd72-4d8b-a533-79334334789f\") " pod="kuadrant-system/authorino-operator-779bff956b-bmwkw" Oct 30 17:56:58.814994 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:58.814969 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-779bff956b-bmwkw"] Oct 30 17:56:58.824312 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:58.824290 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-qgbwm\"" Oct 30 17:56:58.904414 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:58.904379 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h284\" (UniqueName: \"kubernetes.io/projected/8e4d9f9b-bd72-4d8b-a533-79334334789f-kube-api-access-8h284\") pod \"authorino-operator-779bff956b-bmwkw\" (UID: \"8e4d9f9b-bd72-4d8b-a533-79334334789f\") " pod="kuadrant-system/authorino-operator-779bff956b-bmwkw" Oct 30 17:56:58.935709 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:58.935683 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h284\" (UniqueName: \"kubernetes.io/projected/8e4d9f9b-bd72-4d8b-a533-79334334789f-kube-api-access-8h284\") pod \"authorino-operator-779bff956b-bmwkw\" (UID: \"8e4d9f9b-bd72-4d8b-a533-79334334789f\") " pod="kuadrant-system/authorino-operator-779bff956b-bmwkw" Oct 30 17:56:58.975008 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:58.974979 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76d5f9c5c-fb457" Oct 30 17:56:59.022849 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:59.022813 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-57f5c66455-l7hkg" event={"ID":"64c72761-8ae7-47a1-aa47-d10cadeb3e34","Type":"ContainerStarted","Data":"031530840e04adce4b543234c2a78e739f5643dcc9ebceb3de03381970c76089"} Oct 30 17:56:59.106515 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:59.106433 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-779bff956b-bmwkw" Oct 30 17:56:59.312885 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:59.312842 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b575997cc-27c49"] Oct 30 17:56:59.315649 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:59.315624 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b575997cc-27c49" Oct 30 17:56:59.352621 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:59.351487 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b575997cc-27c49"] Oct 30 17:56:59.409118 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:59.408989 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-779bff956b-bmwkw"] Oct 30 17:56:59.409799 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:59.409769 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whzh7\" (UniqueName: \"kubernetes.io/projected/2cb9c001-fc99-4e94-908f-e3a59edeb3ab-kube-api-access-whzh7\") pod \"kuadrant-operator-controller-manager-6b575997cc-27c49\" (UID: \"2cb9c001-fc99-4e94-908f-e3a59edeb3ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b575997cc-27c49" Oct 30 17:56:59.412682 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:56:59.412654 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e4d9f9b_bd72_4d8b_a533_79334334789f.slice/crio-a21a6948d90a9eace5634214bfed4c87a93a3bb5d4fd0f29c62b1e7d38aa480d WatchSource:0}: Error finding container a21a6948d90a9eace5634214bfed4c87a93a3bb5d4fd0f29c62b1e7d38aa480d: Status 404 returned error can't find the container with id a21a6948d90a9eace5634214bfed4c87a93a3bb5d4fd0f29c62b1e7d38aa480d Oct 30 17:56:59.510454 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:59.510411 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whzh7\" (UniqueName: \"kubernetes.io/projected/2cb9c001-fc99-4e94-908f-e3a59edeb3ab-kube-api-access-whzh7\") pod \"kuadrant-operator-controller-manager-6b575997cc-27c49\" (UID: \"2cb9c001-fc99-4e94-908f-e3a59edeb3ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b575997cc-27c49" Oct 30 17:56:59.535929 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:59.535900 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whzh7\" (UniqueName: \"kubernetes.io/projected/2cb9c001-fc99-4e94-908f-e3a59edeb3ab-kube-api-access-whzh7\") pod \"kuadrant-operator-controller-manager-6b575997cc-27c49\" (UID: \"2cb9c001-fc99-4e94-908f-e3a59edeb3ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b575997cc-27c49" Oct 30 17:56:59.633664 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:59.633629 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b575997cc-27c49" Oct 30 17:56:59.960049 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:56:59.959943 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b575997cc-27c49"] Oct 30 17:56:59.972804 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:56:59.972770 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb9c001_fc99_4e94_908f_e3a59edeb3ab.slice/crio-9ba948351947d0f1a426eb5aa84b02ca0310001ae14c801c56aa48c6c6de0808 WatchSource:0}: Error finding container 9ba948351947d0f1a426eb5aa84b02ca0310001ae14c801c56aa48c6c6de0808: Status 404 returned error can't find the container with id 9ba948351947d0f1a426eb5aa84b02ca0310001ae14c801c56aa48c6c6de0808 Oct 30 17:57:00.032516 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:00.032474 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b575997cc-27c49" event={"ID":"2cb9c001-fc99-4e94-908f-e3a59edeb3ab","Type":"ContainerStarted","Data":"9ba948351947d0f1a426eb5aa84b02ca0310001ae14c801c56aa48c6c6de0808"} Oct 30 17:57:00.033907 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:00.033877 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-779bff956b-bmwkw" event={"ID":"8e4d9f9b-bd72-4d8b-a533-79334334789f","Type":"ContainerStarted","Data":"a21a6948d90a9eace5634214bfed4c87a93a3bb5d4fd0f29c62b1e7d38aa480d"} Oct 30 17:57:00.387240 ip-10-0-130-84 kubenswrapper[4455]: E1030 17:57:00.387189 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 17:57:01.044223 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:01.044174 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b575997cc-27c49" event={"ID":"2cb9c001-fc99-4e94-908f-e3a59edeb3ab","Type":"ContainerStarted","Data":"98b694eaf6a4f34361e664b72b1495f6f4a585438ff8143f007eeead686ef121"} Oct 30 17:57:01.044758 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:01.044449 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b575997cc-27c49" Oct 30 17:57:01.116304 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:01.116236 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b575997cc-27c49" podStartSLOduration=2.11621696 podStartE2EDuration="2.11621696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:01.113970352 +0000 UTC m=+273.601607389" watchObservedRunningTime="2026-10-30 17:57:01.11621696 +0000 UTC m=+273.603853992" Oct 30 17:57:02.050150 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:02.050118 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-779bff956b-bmwkw" event={"ID":"8e4d9f9b-bd72-4d8b-a533-79334334789f","Type":"ContainerStarted","Data":"7e4f0354bd90969fe84c668d50e9d56786f676303f880abb9f4f96f330aa53d5"} Oct 30 17:57:02.050634 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:02.050293 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-779bff956b-bmwkw" Oct 30 17:57:02.052305 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:02.052273 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-57f5c66455-l7hkg" event={"ID":"64c72761-8ae7-47a1-aa47-d10cadeb3e34","Type":"ContainerStarted","Data":"2b69b01b85480b0c4b7f5bf17beb57118c3dde303b369430700f8d0b1ea02ec0"} Oct 30 17:57:02.052448 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:02.052438 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-57f5c66455-l7hkg" Oct 30 17:57:02.090692 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:02.090645 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-779bff956b-bmwkw" podStartSLOduration=1.884751639 podStartE2EDuration="4.0906318s" totalImagesPullingTime="2.205880161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:02.089136342 +0000 UTC m=+274.576773375" watchObservedRunningTime="2026-10-30 17:57:02.0906318 +0000 UTC m=+274.578268835" Oct 30 17:57:02.134065 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:02.134014 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-57f5c66455-l7hkg" podStartSLOduration=2.134374116 podStartE2EDuration="5.134001668s" totalImagesPullingTime="2.999627552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:02.132262778 +0000 UTC m=+274.619899812" watchObservedRunningTime="2026-10-30 17:57:02.134001668 +0000 UTC m=+274.621638703" Oct 30 17:57:03.950520 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:03.950485 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44"] Oct 30 17:57:03.953225 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:03.953206 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" Oct 30 17:57:03.973671 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:03.973649 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-fpjjp\"" Oct 30 17:57:03.987601 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:03.987575 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44"] Oct 30 17:57:04.047408 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:04.047375 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae819cc-7f64-4b4a-8559-1bc35afe67d1-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44\" (UID: \"5ae819cc-7f64-4b4a-8559-1bc35afe67d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" Oct 30 17:57:04.047577 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:04.047428 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ae819cc-7f64-4b4a-8559-1bc35afe67d1-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44\" (UID: \"5ae819cc-7f64-4b4a-8559-1bc35afe67d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" Oct 30 17:57:04.047577 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:04.047532 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae819cc-7f64-4b4a-8559-1bc35afe67d1-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44\" (UID: \"5ae819cc-7f64-4b4a-8559-1bc35afe67d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" Oct 30 17:57:04.047577 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:04.047572 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ae819cc-7f64-4b4a-8559-1bc35afe67d1-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44\" (UID: \"5ae819cc-7f64-4b4a-8559-1bc35afe67d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" Oct 30 17:57:04.047680 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:04.047650 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ae819cc-7f64-4b4a-8559-1bc35afe67d1-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44\" (UID: \"5ae819cc-7f64-4b4a-8559-1bc35afe67d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" Oct 30 17:57:04.047715 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:04.047700 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ae819cc-7f64-4b4a-8559-1bc35afe67d1-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44\" (UID: \"5ae819cc-7f64-4b4a-8559-1bc35afe67d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" Oct 30 17:57:04.047751 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:04.047736 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ae819cc-7f64-4b4a-8559-1bc35afe67d1-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44\" (UID: \"5ae819cc-7f64-4b4a-8559-1bc35afe67d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" Oct 30 17:57:04.047793 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:04.047783 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ae819cc-7f64-4b4a-8559-1bc35afe67d1-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44\" (UID: \"5ae819cc-7f64-4b4a-8559-1bc35afe67d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" Oct 30 17:57:04.047828 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:04.047815 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl972\" (UniqueName: \"kubernetes.io/projected/5ae819cc-7f64-4b4a-8559-1bc35afe67d1-kube-api-access-cl972\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44\" (UID: \"5ae819cc-7f64-4b4a-8559-1bc35afe67d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" Oct 30 17:57:04.148983 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:04.148946 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae819cc-7f64-4b4a-8559-1bc35afe67d1-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44\" (UID: \"5ae819cc-7f64-4b4a-8559-1bc35afe67d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" Oct 30 17:57:04.148983 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:04.148985 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ae819cc-7f64-4b4a-8559-1bc35afe67d1-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44\" (UID: \"5ae819cc-7f64-4b4a-8559-1bc35afe67d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" Oct 30 17:57:04.149234 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:04.149123 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ae819cc-7f64-4b4a-8559-1bc35afe67d1-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44\" (UID: \"5ae819cc-7f64-4b4a-8559-1bc35afe67d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" Oct 30 17:57:04.149234 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:04.149215 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ae819cc-7f64-4b4a-8559-1bc35afe67d1-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44\" (UID: \"5ae819cc-7f64-4b4a-8559-1bc35afe67d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" Oct 30 17:57:04.149372 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:04.149245 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ae819cc-7f64-4b4a-8559-1bc35afe67d1-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44\" (UID: \"5ae819cc-7f64-4b4a-8559-1bc35afe67d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" Oct 30 17:57:04.149372 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:04.149282 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ae819cc-7f64-4b4a-8559-1bc35afe67d1-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44\" (UID: \"5ae819cc-7f64-4b4a-8559-1bc35afe67d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" Oct 30 17:57:04.149372 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:04.149309 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl972\" (UniqueName: \"kubernetes.io/projected/5ae819cc-7f64-4b4a-8559-1bc35afe67d1-kube-api-access-cl972\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44\" (UID: \"5ae819cc-7f64-4b4a-8559-1bc35afe67d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" Oct 30 17:57:04.149533 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:04.149387 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae819cc-7f64-4b4a-8559-1bc35afe67d1-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44\" (UID: \"5ae819cc-7f64-4b4a-8559-1bc35afe67d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" Oct 30 17:57:04.149533 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:04.149438 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ae819cc-7f64-4b4a-8559-1bc35afe67d1-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44\" (UID: \"5ae819cc-7f64-4b4a-8559-1bc35afe67d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" Oct 30 17:57:04.149661 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:04.149637 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ae819cc-7f64-4b4a-8559-1bc35afe67d1-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44\" (UID: \"5ae819cc-7f64-4b4a-8559-1bc35afe67d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" Oct 30 17:57:04.149753 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:04.149731 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ae819cc-7f64-4b4a-8559-1bc35afe67d1-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44\" (UID: \"5ae819cc-7f64-4b4a-8559-1bc35afe67d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" Oct 30 17:57:04.149814 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:04.149766 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae819cc-7f64-4b4a-8559-1bc35afe67d1-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44\" (UID: \"5ae819cc-7f64-4b4a-8559-1bc35afe67d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" Oct 30 17:57:04.149967 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:04.149947 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae819cc-7f64-4b4a-8559-1bc35afe67d1-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44\" (UID: \"5ae819cc-7f64-4b4a-8559-1bc35afe67d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" Oct 30 17:57:04.150014 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:04.149975 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ae819cc-7f64-4b4a-8559-1bc35afe67d1-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44\" (UID: \"5ae819cc-7f64-4b4a-8559-1bc35afe67d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" Oct 30 17:57:04.151452 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:04.151427 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ae819cc-7f64-4b4a-8559-1bc35afe67d1-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44\" (UID: \"5ae819cc-7f64-4b4a-8559-1bc35afe67d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" Oct 30 17:57:04.152014 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:04.151997 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ae819cc-7f64-4b4a-8559-1bc35afe67d1-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44\" (UID: \"5ae819cc-7f64-4b4a-8559-1bc35afe67d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" Oct 30 17:57:04.172230 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:04.172208 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ae819cc-7f64-4b4a-8559-1bc35afe67d1-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44\" (UID: \"5ae819cc-7f64-4b4a-8559-1bc35afe67d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" Oct 30 17:57:04.172562 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:04.172546 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl972\" (UniqueName: \"kubernetes.io/projected/5ae819cc-7f64-4b4a-8559-1bc35afe67d1-kube-api-access-cl972\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44\" (UID: \"5ae819cc-7f64-4b4a-8559-1bc35afe67d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" Oct 30 17:57:04.264970 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:04.264940 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" Oct 30 17:57:04.577098 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:04.577070 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44"] Oct 30 17:57:04.579184 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:57:04.579152 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae819cc_7f64_4b4a_8559_1bc35afe67d1.slice/crio-ddc0566a8474e58fe2037784721db3c272a83a71abff85bf8b67892564c10f3e WatchSource:0}: Error finding container ddc0566a8474e58fe2037784721db3c272a83a71abff85bf8b67892564c10f3e: Status 404 returned error can't find the container with id ddc0566a8474e58fe2037784721db3c272a83a71abff85bf8b67892564c10f3e Oct 30 17:57:04.581381 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:04.581348 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 17:57:04.581455 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:04.581413 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 17:57:04.581455 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:04.581445 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 17:57:05.069171 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:05.069130 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" event={"ID":"5ae819cc-7f64-4b4a-8559-1bc35afe67d1","Type":"ContainerStarted","Data":"7e85f6f4d8bbc5a74c833f99d48b119d6b92dbe34d1dddc0c0b89c45816f49ac"} Oct 30 17:57:05.069604 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:05.069183 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" event={"ID":"5ae819cc-7f64-4b4a-8559-1bc35afe67d1","Type":"ContainerStarted","Data":"ddc0566a8474e58fe2037784721db3c272a83a71abff85bf8b67892564c10f3e"} Oct 30 17:57:05.112096 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:05.112050 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" podStartSLOduration=2.112035361 podStartE2EDuration="2.112035361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:05.11192208 +0000 UTC m=+277.599559115" watchObservedRunningTime="2026-10-30 17:57:05.112035361 +0000 UTC m=+277.599672412" Oct 30 17:57:05.265592 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:05.265559 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" Oct 30 17:57:05.272178 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:05.272149 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" Oct 30 17:57:06.074588 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:06.074560 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" Oct 30 17:57:06.075638 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:06.075617 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44" Oct 30 17:57:06.355684 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:06.355612 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-74ccc46849-m6vxp"] Oct 30 17:57:06.429569 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:06.429537 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74ccc46849-m6vxp"] Oct 30 17:57:06.429719 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:06.429630 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74ccc46849-m6vxp" Oct 30 17:57:06.445078 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:06.445059 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-fnvtk\"" Oct 30 17:57:06.471748 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:06.471722 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5vd6\" (UniqueName: \"kubernetes.io/projected/b498d263-fe59-4771-82af-ef277f4d51a4-kube-api-access-z5vd6\") pod \"authorino-74ccc46849-m6vxp\" (UID: \"b498d263-fe59-4771-82af-ef277f4d51a4\") " pod="kuadrant-system/authorino-74ccc46849-m6vxp" Oct 30 17:57:06.572353 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:06.572305 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5vd6\" (UniqueName: \"kubernetes.io/projected/b498d263-fe59-4771-82af-ef277f4d51a4-kube-api-access-z5vd6\") pod \"authorino-74ccc46849-m6vxp\" (UID: \"b498d263-fe59-4771-82af-ef277f4d51a4\") " pod="kuadrant-system/authorino-74ccc46849-m6vxp" Oct 30 17:57:06.594876 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:06.594848 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5vd6\" (UniqueName: \"kubernetes.io/projected/b498d263-fe59-4771-82af-ef277f4d51a4-kube-api-access-z5vd6\") pod \"authorino-74ccc46849-m6vxp\" (UID: \"b498d263-fe59-4771-82af-ef277f4d51a4\") " pod="kuadrant-system/authorino-74ccc46849-m6vxp" Oct 30 17:57:06.756637 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:06.756602 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74ccc46849-m6vxp" Oct 30 17:57:07.050024 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:07.049997 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74ccc46849-m6vxp"] Oct 30 17:57:07.059069 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:57:07.059036 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb498d263_fe59_4771_82af_ef277f4d51a4.slice/crio-d3bba4aa0a5cd3c904853b30b063cb6bcf268ed7c9054ee74f31432917515c6a WatchSource:0}: Error finding container d3bba4aa0a5cd3c904853b30b063cb6bcf268ed7c9054ee74f31432917515c6a: Status 404 returned error can't find the container with id d3bba4aa0a5cd3c904853b30b063cb6bcf268ed7c9054ee74f31432917515c6a Oct 30 17:57:07.080132 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:07.080099 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74ccc46849-m6vxp" event={"ID":"b498d263-fe59-4771-82af-ef277f4d51a4","Type":"ContainerStarted","Data":"d3bba4aa0a5cd3c904853b30b063cb6bcf268ed7c9054ee74f31432917515c6a"} Oct 30 17:57:08.392867 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:08.392831 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-5tcnq"] Oct 30 17:57:08.395859 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:08.395835 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tcnq" Oct 30 17:57:08.410401 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:08.410378 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 17:57:08.421865 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:08.421845 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-mf782\"" Oct 30 17:57:08.428070 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:08.428043 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-5tcnq"] Oct 30 17:57:08.488396 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:08.488357 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0d007217-7dd2-4980-a42c-8e4df31e4293-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-5tcnq\" (UID: \"0d007217-7dd2-4980-a42c-8e4df31e4293\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tcnq" Oct 30 17:57:08.488578 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:08.488422 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0d007217-7dd2-4980-a42c-8e4df31e4293-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-5tcnq\" (UID: \"0d007217-7dd2-4980-a42c-8e4df31e4293\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tcnq" Oct 30 17:57:08.488664 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:08.488606 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql9fg\" (UniqueName: \"kubernetes.io/projected/0d007217-7dd2-4980-a42c-8e4df31e4293-kube-api-access-ql9fg\") pod \"kserve-module-controller-manager-7566cf5fc8-5tcnq\" (UID: \"0d007217-7dd2-4980-a42c-8e4df31e4293\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tcnq" Oct 30 17:57:08.488722 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:08.488703 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d007217-7dd2-4980-a42c-8e4df31e4293-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-5tcnq\" (UID: \"0d007217-7dd2-4980-a42c-8e4df31e4293\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tcnq" Oct 30 17:57:08.516163 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:08.516134 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 17:57:08.528808 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:08.528777 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 17:57:08.590143 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:08.590111 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0d007217-7dd2-4980-a42c-8e4df31e4293-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-5tcnq\" (UID: \"0d007217-7dd2-4980-a42c-8e4df31e4293\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tcnq" Oct 30 17:57:08.590347 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:08.590177 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql9fg\" (UniqueName: \"kubernetes.io/projected/0d007217-7dd2-4980-a42c-8e4df31e4293-kube-api-access-ql9fg\") pod \"kserve-module-controller-manager-7566cf5fc8-5tcnq\" (UID: \"0d007217-7dd2-4980-a42c-8e4df31e4293\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tcnq" Oct 30 17:57:08.590347 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:08.590216 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d007217-7dd2-4980-a42c-8e4df31e4293-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-5tcnq\" (UID: \"0d007217-7dd2-4980-a42c-8e4df31e4293\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tcnq" Oct 30 17:57:08.590347 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:08.590287 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0d007217-7dd2-4980-a42c-8e4df31e4293-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-5tcnq\" (UID: \"0d007217-7dd2-4980-a42c-8e4df31e4293\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tcnq" Oct 30 17:57:08.590721 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:08.590702 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0d007217-7dd2-4980-a42c-8e4df31e4293-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-5tcnq\" (UID: \"0d007217-7dd2-4980-a42c-8e4df31e4293\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tcnq" Oct 30 17:57:08.590766 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:08.590719 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d007217-7dd2-4980-a42c-8e4df31e4293-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-5tcnq\" (UID: \"0d007217-7dd2-4980-a42c-8e4df31e4293\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tcnq" Oct 30 17:57:08.590994 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:08.590966 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0d007217-7dd2-4980-a42c-8e4df31e4293-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-5tcnq\" (UID: \"0d007217-7dd2-4980-a42c-8e4df31e4293\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tcnq" Oct 30 17:57:08.613572 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:08.613543 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql9fg\" (UniqueName: \"kubernetes.io/projected/0d007217-7dd2-4980-a42c-8e4df31e4293-kube-api-access-ql9fg\") pod \"kserve-module-controller-manager-7566cf5fc8-5tcnq\" (UID: \"0d007217-7dd2-4980-a42c-8e4df31e4293\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tcnq" Oct 30 17:57:08.708504 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:08.708428 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tcnq" Oct 30 17:57:09.024999 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:09.024974 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-5tcnq"] Oct 30 17:57:09.027221 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:57:09.027190 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d007217_7dd2_4980_a42c_8e4df31e4293.slice/crio-161c8450bf42499d97e1285e8c88be58fb88f2a84cddf4a3eaecb42bd0b794c3 WatchSource:0}: Error finding container 161c8450bf42499d97e1285e8c88be58fb88f2a84cddf4a3eaecb42bd0b794c3: Status 404 returned error can't find the container with id 161c8450bf42499d97e1285e8c88be58fb88f2a84cddf4a3eaecb42bd0b794c3 Oct 30 17:57:09.092367 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:09.092308 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tcnq" event={"ID":"0d007217-7dd2-4980-a42c-8e4df31e4293","Type":"ContainerStarted","Data":"161c8450bf42499d97e1285e8c88be58fb88f2a84cddf4a3eaecb42bd0b794c3"} Oct 30 17:57:10.102175 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:10.102129 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74ccc46849-m6vxp" event={"ID":"b498d263-fe59-4771-82af-ef277f4d51a4","Type":"ContainerStarted","Data":"949fca8130f7f644b2b7a54529d6dfb3b1b855063e5b0719ae7244a842db15b2"} Oct 30 17:57:10.545830 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:10.545773 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-74ccc46849-m6vxp" podStartSLOduration=2.048917281 podStartE2EDuration="4.545757003s" totalImagesPullingTime="2.496839722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:10.139900873 +0000 UTC m=+282.627537907" watchObservedRunningTime="2026-10-30 17:57:10.545757003 +0000 UTC m=+283.033394037" Oct 30 17:57:10.546910 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:10.546873 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-74ccc46849-m6vxp"] Oct 30 17:57:12.054765 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:12.054735 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b575997cc-27c49" Oct 30 17:57:12.114791 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:12.114756 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tcnq" event={"ID":"0d007217-7dd2-4980-a42c-8e4df31e4293","Type":"ContainerStarted","Data":"2556fe553a8306d3bc546cfa7ba878bfd32498ab7d40d10fd6a28cef24eaa119"} Oct 30 17:57:12.114791 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:12.114795 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-74ccc46849-m6vxp" podUID="b498d263-fe59-4771-82af-ef277f4d51a4" containerName="authorino" containerID="cri-o://949fca8130f7f644b2b7a54529d6dfb3b1b855063e5b0719ae7244a842db15b2" gracePeriod=30 Oct 30 17:57:12.145983 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:12.145949 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76d5f9c5c-fb457"] Oct 30 17:57:12.146371 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:12.146307 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76d5f9c5c-fb457" podUID="a896d805-773f-4bb0-a8ab-36dfec29af5c" containerName="manager" containerID="cri-o://bd9aba662a0a97d94763267ed6e67a09284d902ad926ca2bc591df19b1c87644" gracePeriod=10 Oct 30 17:57:12.202566 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:12.202520 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tcnq" podStartSLOduration=1.806849261 podStartE2EDuration="4.202506509s" totalImagesPullingTime="2.395657248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:12.20092126 +0000 UTC m=+284.688558285" watchObservedRunningTime="2026-10-30 17:57:12.202506509 +0000 UTC m=+284.690143571" Oct 30 17:57:12.392350 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:12.392314 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74ccc46849-m6vxp" Oct 30 17:57:12.413699 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:12.413675 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76d5f9c5c-fb457" Oct 30 17:57:12.431012 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:12.430989 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b498d263-fe59-4771-82af-ef277f4d51a4-kube-api-access-z5vd6\" (UniqueName: \"kubernetes.io/projected/b498d263-fe59-4771-82af-ef277f4d51a4-kube-api-access-z5vd6\") pod \"b498d263-fe59-4771-82af-ef277f4d51a4\" (UID: \"b498d263-fe59-4771-82af-ef277f4d51a4\") " Oct 30 17:57:12.433206 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:12.433178 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b498d263-fe59-4771-82af-ef277f4d51a4-kube-api-access-z5vd6" pod "b498d263-fe59-4771-82af-ef277f4d51a4" (UID: "b498d263-fe59-4771-82af-ef277f4d51a4"). InnerVolumeSpecName "kube-api-access-z5vd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:57:12.532361 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:12.532333 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a896d805-773f-4bb0-a8ab-36dfec29af5c-kube-api-access-cwkch\" (UniqueName: \"kubernetes.io/projected/a896d805-773f-4bb0-a8ab-36dfec29af5c-kube-api-access-cwkch\") pod \"a896d805-773f-4bb0-a8ab-36dfec29af5c\" (UID: \"a896d805-773f-4bb0-a8ab-36dfec29af5c\") " Oct 30 17:57:12.532618 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:12.532590 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5vd6\" (UniqueName: \"kubernetes.io/projected/b498d263-fe59-4771-82af-ef277f4d51a4-kube-api-access-z5vd6\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:57:12.534491 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:12.534464 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a896d805-773f-4bb0-a8ab-36dfec29af5c-kube-api-access-cwkch" pod "a896d805-773f-4bb0-a8ab-36dfec29af5c" (UID: "a896d805-773f-4bb0-a8ab-36dfec29af5c"). InnerVolumeSpecName "kube-api-access-cwkch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:57:12.633093 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:12.633021 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwkch\" (UniqueName: \"kubernetes.io/projected/a896d805-773f-4bb0-a8ab-36dfec29af5c-kube-api-access-cwkch\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:57:13.060106 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:13.060078 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-779bff956b-bmwkw" Oct 30 17:57:13.060525 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:13.060150 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-57f5c66455-l7hkg" Oct 30 17:57:13.122570 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:13.122533 4455 generic.go:356] "Generic (PLEG): container finished" podID="b498d263-fe59-4771-82af-ef277f4d51a4" containerID="949fca8130f7f644b2b7a54529d6dfb3b1b855063e5b0719ae7244a842db15b2" exitCode=0 Oct 30 17:57:13.122746 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:13.122581 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74ccc46849-m6vxp" Oct 30 17:57:13.122746 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:13.122618 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74ccc46849-m6vxp" event={"ID":"b498d263-fe59-4771-82af-ef277f4d51a4","Type":"ContainerDied","Data":"949fca8130f7f644b2b7a54529d6dfb3b1b855063e5b0719ae7244a842db15b2"} Oct 30 17:57:13.122746 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:13.122675 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74ccc46849-m6vxp" event={"ID":"b498d263-fe59-4771-82af-ef277f4d51a4","Type":"ContainerDied","Data":"d3bba4aa0a5cd3c904853b30b063cb6bcf268ed7c9054ee74f31432917515c6a"} Oct 30 17:57:13.122746 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:13.122696 4455 scope.go:122] "RemoveContainer" containerID="949fca8130f7f644b2b7a54529d6dfb3b1b855063e5b0719ae7244a842db15b2" Oct 30 17:57:13.129440 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:13.129410 4455 generic.go:356] "Generic (PLEG): container finished" podID="a896d805-773f-4bb0-a8ab-36dfec29af5c" containerID="bd9aba662a0a97d94763267ed6e67a09284d902ad926ca2bc591df19b1c87644" exitCode=0 Oct 30 17:57:13.129577 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:13.129472 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76d5f9c5c-fb457" Oct 30 17:57:13.129577 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:13.129504 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76d5f9c5c-fb457" event={"ID":"a896d805-773f-4bb0-a8ab-36dfec29af5c","Type":"ContainerDied","Data":"bd9aba662a0a97d94763267ed6e67a09284d902ad926ca2bc591df19b1c87644"} Oct 30 17:57:13.129577 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:13.129567 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76d5f9c5c-fb457" event={"ID":"a896d805-773f-4bb0-a8ab-36dfec29af5c","Type":"ContainerDied","Data":"38a26c93d08299a355b30bb4a63f5783a9b34a4b8161be5c510ea0737ac5dfd0"} Oct 30 17:57:13.130092 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:13.129840 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tcnq" Oct 30 17:57:13.133091 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:13.133077 4455 scope.go:122] "RemoveContainer" containerID="949fca8130f7f644b2b7a54529d6dfb3b1b855063e5b0719ae7244a842db15b2" Oct 30 17:57:13.133440 ip-10-0-130-84 kubenswrapper[4455]: E1030 17:57:13.133421 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"949fca8130f7f644b2b7a54529d6dfb3b1b855063e5b0719ae7244a842db15b2\": container with ID starting with 949fca8130f7f644b2b7a54529d6dfb3b1b855063e5b0719ae7244a842db15b2 not found: ID does not exist" containerID="949fca8130f7f644b2b7a54529d6dfb3b1b855063e5b0719ae7244a842db15b2" Oct 30 17:57:13.133508 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:13.133449 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"949fca8130f7f644b2b7a54529d6dfb3b1b855063e5b0719ae7244a842db15b2"} err="failed to get container status \"949fca8130f7f644b2b7a54529d6dfb3b1b855063e5b0719ae7244a842db15b2\": rpc error: code = NotFound desc = could not find container \"949fca8130f7f644b2b7a54529d6dfb3b1b855063e5b0719ae7244a842db15b2\": container with ID starting with 949fca8130f7f644b2b7a54529d6dfb3b1b855063e5b0719ae7244a842db15b2 not found: ID does not exist" Oct 30 17:57:13.133508 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:13.133466 4455 scope.go:122] "RemoveContainer" containerID="bd9aba662a0a97d94763267ed6e67a09284d902ad926ca2bc591df19b1c87644" Oct 30 17:57:13.143783 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:13.143751 4455 scope.go:122] "RemoveContainer" containerID="bd9aba662a0a97d94763267ed6e67a09284d902ad926ca2bc591df19b1c87644" Oct 30 17:57:13.144139 ip-10-0-130-84 kubenswrapper[4455]: E1030 17:57:13.144115 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd9aba662a0a97d94763267ed6e67a09284d902ad926ca2bc591df19b1c87644\": container with ID starting with bd9aba662a0a97d94763267ed6e67a09284d902ad926ca2bc591df19b1c87644 not found: ID does not exist" containerID="bd9aba662a0a97d94763267ed6e67a09284d902ad926ca2bc591df19b1c87644" Oct 30 17:57:13.144215 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:13.144148 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd9aba662a0a97d94763267ed6e67a09284d902ad926ca2bc591df19b1c87644"} err="failed to get container status \"bd9aba662a0a97d94763267ed6e67a09284d902ad926ca2bc591df19b1c87644\": rpc error: code = NotFound desc = could not find container \"bd9aba662a0a97d94763267ed6e67a09284d902ad926ca2bc591df19b1c87644\": container with ID starting with bd9aba662a0a97d94763267ed6e67a09284d902ad926ca2bc591df19b1c87644 not found: ID does not exist" Oct 30 17:57:13.186286 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:13.186259 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76d5f9c5c-fb457"] Oct 30 17:57:13.198535 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:13.198509 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76d5f9c5c-fb457"] Oct 30 17:57:13.227183 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:13.227155 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-74ccc46849-m6vxp"] Oct 30 17:57:13.233404 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:13.233385 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-74ccc46849-m6vxp"] Oct 30 17:57:13.632141 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:13.632109 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a896d805-773f-4bb0-a8ab-36dfec29af5c" path="/var/lib/kubelet/pods/a896d805-773f-4bb0-a8ab-36dfec29af5c/volumes" Oct 30 17:57:13.632456 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:13.632443 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b498d263-fe59-4771-82af-ef277f4d51a4" path="/var/lib/kubelet/pods/b498d263-fe59-4771-82af-ef277f4d51a4/volumes" Oct 30 17:57:17.344208 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:17.344169 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-tgr57"] Oct 30 17:57:17.344893 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:17.344865 4455 cpu_manager.go:414] "containerMap: removing container" podUID="b498d263-fe59-4771-82af-ef277f4d51a4" containerName="authorino" Oct 30 17:57:17.344893 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:17.344890 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b498d263-fe59-4771-82af-ef277f4d51a4" containerName="authorino" Oct 30 17:57:17.345046 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:17.344913 4455 cpu_manager.go:414] "containerMap: removing container" podUID="a896d805-773f-4bb0-a8ab-36dfec29af5c" containerName="manager" Oct 30 17:57:17.345046 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:17.344918 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a896d805-773f-4bb0-a8ab-36dfec29af5c" containerName="manager" Oct 30 17:57:17.345046 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:17.344994 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="b498d263-fe59-4771-82af-ef277f4d51a4" containerName="authorino" Oct 30 17:57:17.345046 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:17.345006 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="a896d805-773f-4bb0-a8ab-36dfec29af5c" containerName="manager" Oct 30 17:57:17.369651 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:17.369625 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-tgr57" Oct 30 17:57:17.373441 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:17.373415 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-tgr57"] Oct 30 17:57:17.384558 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:17.384536 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 17:57:17.395611 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:17.395589 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 17:57:17.395611 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:17.395607 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-qg77h\"" Oct 30 17:57:17.471728 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:17.471687 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwfgg\" (UniqueName: \"kubernetes.io/projected/a22891f5-1a57-41df-b7f2-09a44c920912-kube-api-access-bwfgg\") pod \"cert-manager-597b95cb84-tgr57\" (UID: \"a22891f5-1a57-41df-b7f2-09a44c920912\") " pod="cert-manager/cert-manager-597b95cb84-tgr57" Oct 30 17:57:17.471882 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:17.471735 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a22891f5-1a57-41df-b7f2-09a44c920912-bound-sa-token\") pod \"cert-manager-597b95cb84-tgr57\" (UID: \"a22891f5-1a57-41df-b7f2-09a44c920912\") " pod="cert-manager/cert-manager-597b95cb84-tgr57" Oct 30 17:57:17.573362 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:17.573298 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwfgg\" (UniqueName: \"kubernetes.io/projected/a22891f5-1a57-41df-b7f2-09a44c920912-kube-api-access-bwfgg\") pod \"cert-manager-597b95cb84-tgr57\" (UID: \"a22891f5-1a57-41df-b7f2-09a44c920912\") " pod="cert-manager/cert-manager-597b95cb84-tgr57" Oct 30 17:57:17.573362 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:17.573368 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a22891f5-1a57-41df-b7f2-09a44c920912-bound-sa-token\") pod \"cert-manager-597b95cb84-tgr57\" (UID: \"a22891f5-1a57-41df-b7f2-09a44c920912\") " pod="cert-manager/cert-manager-597b95cb84-tgr57" Oct 30 17:57:17.596804 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:17.596736 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwfgg\" (UniqueName: \"kubernetes.io/projected/a22891f5-1a57-41df-b7f2-09a44c920912-kube-api-access-bwfgg\") pod \"cert-manager-597b95cb84-tgr57\" (UID: \"a22891f5-1a57-41df-b7f2-09a44c920912\") " pod="cert-manager/cert-manager-597b95cb84-tgr57" Oct 30 17:57:17.596804 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:17.596746 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a22891f5-1a57-41df-b7f2-09a44c920912-bound-sa-token\") pod \"cert-manager-597b95cb84-tgr57\" (UID: \"a22891f5-1a57-41df-b7f2-09a44c920912\") " pod="cert-manager/cert-manager-597b95cb84-tgr57" Oct 30 17:57:17.690532 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:17.690502 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-tgr57" Oct 30 17:57:18.002700 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:18.002656 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-tgr57"] Oct 30 17:57:18.004921 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:57:18.004895 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda22891f5_1a57_41df_b7f2_09a44c920912.slice/crio-b7264d21698a1060bbc719a41f74c5ebc388905fc4b8140d55b87164a8d5d805 WatchSource:0}: Error finding container b7264d21698a1060bbc719a41f74c5ebc388905fc4b8140d55b87164a8d5d805: Status 404 returned error can't find the container with id b7264d21698a1060bbc719a41f74c5ebc388905fc4b8140d55b87164a8d5d805 Oct 30 17:57:18.160643 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:18.160609 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-tgr57" event={"ID":"a22891f5-1a57-41df-b7f2-09a44c920912","Type":"ContainerStarted","Data":"b7264d21698a1060bbc719a41f74c5ebc388905fc4b8140d55b87164a8d5d805"} Oct 30 17:57:21.178522 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:21.178438 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-tgr57" event={"ID":"a22891f5-1a57-41df-b7f2-09a44c920912","Type":"ContainerStarted","Data":"7a1bafb0edcfbfb15d0ea5d0be841c8fd66dfaace2771d53eba52df19c5391d1"} Oct 30 17:57:21.219664 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:21.219510 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-tgr57" podStartSLOduration=1.752317323 podStartE2EDuration="4.219491629s" totalImagesPullingTime="2.467174306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:21.21753851 +0000 UTC m=+293.705175546" watchObservedRunningTime="2026-10-30 17:57:21.219491629 +0000 UTC m=+293.707128665" Oct 30 17:57:30.384951 ip-10-0-130-84 kubenswrapper[4455]: E1030 17:57:30.384903 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 17:57:34.139020 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:34.138992 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tcnq" Oct 30 17:57:36.460367 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:36.459262 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-fh9zp"] Oct 30 17:57:36.481291 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:36.481265 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fh9zp" Oct 30 17:57:36.483552 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:36.483522 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-fh9zp"] Oct 30 17:57:36.506795 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:36.506771 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:57:36.506937 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:36.506838 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:57:36.506937 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:36.506847 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-v8ss9\"" Oct 30 17:57:36.506937 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:36.506869 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 17:57:36.650740 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:36.650705 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plhwp\" (UniqueName: \"kubernetes.io/projected/725c3af8-4989-4b39-8141-5688451ea2bc-kube-api-access-plhwp\") pod \"postgres-7cdbf6dc44-fh9zp\" (UID: \"725c3af8-4989-4b39-8141-5688451ea2bc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fh9zp" Oct 30 17:57:36.650899 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:36.650783 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/725c3af8-4989-4b39-8141-5688451ea2bc-data\") pod \"postgres-7cdbf6dc44-fh9zp\" (UID: \"725c3af8-4989-4b39-8141-5688451ea2bc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fh9zp" Oct 30 17:57:36.752117 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:36.752083 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/725c3af8-4989-4b39-8141-5688451ea2bc-data\") pod \"postgres-7cdbf6dc44-fh9zp\" (UID: \"725c3af8-4989-4b39-8141-5688451ea2bc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fh9zp" Oct 30 17:57:36.752344 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:36.752302 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plhwp\" (UniqueName: \"kubernetes.io/projected/725c3af8-4989-4b39-8141-5688451ea2bc-kube-api-access-plhwp\") pod \"postgres-7cdbf6dc44-fh9zp\" (UID: \"725c3af8-4989-4b39-8141-5688451ea2bc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fh9zp" Oct 30 17:57:36.752594 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:36.752572 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/725c3af8-4989-4b39-8141-5688451ea2bc-data\") pod \"postgres-7cdbf6dc44-fh9zp\" (UID: \"725c3af8-4989-4b39-8141-5688451ea2bc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fh9zp" Oct 30 17:57:36.775673 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:36.775643 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plhwp\" (UniqueName: \"kubernetes.io/projected/725c3af8-4989-4b39-8141-5688451ea2bc-kube-api-access-plhwp\") pod \"postgres-7cdbf6dc44-fh9zp\" (UID: \"725c3af8-4989-4b39-8141-5688451ea2bc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fh9zp" Oct 30 17:57:36.800532 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:36.800499 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fh9zp" Oct 30 17:57:37.093209 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:37.093187 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-fh9zp"] Oct 30 17:57:37.096038 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:57:37.096008 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod725c3af8_4989_4b39_8141_5688451ea2bc.slice/crio-b9bf48377aacf52fe16812eae8b19069dcd0cadc715bc4daecc32ee81ef6f654 WatchSource:0}: Error finding container b9bf48377aacf52fe16812eae8b19069dcd0cadc715bc4daecc32ee81ef6f654: Status 404 returned error can't find the container with id b9bf48377aacf52fe16812eae8b19069dcd0cadc715bc4daecc32ee81ef6f654 Oct 30 17:57:37.097555 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:37.097531 4455 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:57:37.257704 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:37.257667 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fh9zp" event={"ID":"725c3af8-4989-4b39-8141-5688451ea2bc","Type":"ContainerStarted","Data":"b9bf48377aacf52fe16812eae8b19069dcd0cadc715bc4daecc32ee81ef6f654"} Oct 30 17:57:42.290861 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:42.290823 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fh9zp" event={"ID":"725c3af8-4989-4b39-8141-5688451ea2bc","Type":"ContainerStarted","Data":"580459f635c99160803e47a88575df4797f630a45fcb9c809e0de53ff66b702e"} Oct 30 17:57:42.291346 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:42.290931 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fh9zp" Oct 30 17:57:42.331344 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:42.331268 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fh9zp" podStartSLOduration=1.654341305 podStartE2EDuration="6.331254219s" totalImagesPullingTime="4.676912914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:42.329963613 +0000 UTC m=+314.817600647" watchObservedRunningTime="2026-10-30 17:57:42.331254219 +0000 UTC m=+314.818891253" Oct 30 17:57:48.326365 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:57:48.326317 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fh9zp" Oct 30 17:58:00.400654 ip-10-0-130-84 kubenswrapper[4455]: E1030 17:58:00.400609 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 17:58:05.687475 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:05.687441 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-xrlvs"] Oct 30 17:58:05.695101 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:05.695079 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-xrlvs" Oct 30 17:58:05.718100 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:05.718069 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-xrlvs"] Oct 30 17:58:05.729824 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:05.724016 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 17:58:05.729824 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:05.724125 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-qg5s5\"" Oct 30 17:58:05.769413 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:05.769375 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-6kc8h"] Oct 30 17:58:05.773267 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:05.773249 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-6kc8h" Oct 30 17:58:05.800054 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:05.800024 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-6kc8h"] Oct 30 17:58:05.812391 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:05.812369 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 17:58:05.812391 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:05.812384 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-2chvd\"" Oct 30 17:58:05.825001 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:05.824975 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vnn6\" (UniqueName: \"kubernetes.io/projected/5e6cc7fd-dcda-490d-b3d0-661c28f95e1d-kube-api-access-2vnn6\") pod \"model-serving-api-75bd89fdfc-6kc8h\" (UID: \"5e6cc7fd-dcda-490d-b3d0-661c28f95e1d\") " pod="opendatahub/model-serving-api-75bd89fdfc-6kc8h" Oct 30 17:58:05.825123 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:05.825009 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0242f980-e7c2-4799-964d-b33944c19332-cert\") pod \"kserve-controller-manager-776cc66d47-xrlvs\" (UID: \"0242f980-e7c2-4799-964d-b33944c19332\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xrlvs" Oct 30 17:58:05.825123 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:05.825046 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc68d\" (UniqueName: \"kubernetes.io/projected/0242f980-e7c2-4799-964d-b33944c19332-kube-api-access-dc68d\") pod \"kserve-controller-manager-776cc66d47-xrlvs\" (UID: \"0242f980-e7c2-4799-964d-b33944c19332\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xrlvs" Oct 30 17:58:05.825218 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:05.825154 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e6cc7fd-dcda-490d-b3d0-661c28f95e1d-tls-certs\") pod \"model-serving-api-75bd89fdfc-6kc8h\" (UID: \"5e6cc7fd-dcda-490d-b3d0-661c28f95e1d\") " pod="opendatahub/model-serving-api-75bd89fdfc-6kc8h" Oct 30 17:58:05.926255 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:05.926215 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vnn6\" (UniqueName: \"kubernetes.io/projected/5e6cc7fd-dcda-490d-b3d0-661c28f95e1d-kube-api-access-2vnn6\") pod \"model-serving-api-75bd89fdfc-6kc8h\" (UID: \"5e6cc7fd-dcda-490d-b3d0-661c28f95e1d\") " pod="opendatahub/model-serving-api-75bd89fdfc-6kc8h" Oct 30 17:58:05.926499 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:05.926261 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0242f980-e7c2-4799-964d-b33944c19332-cert\") pod \"kserve-controller-manager-776cc66d47-xrlvs\" (UID: \"0242f980-e7c2-4799-964d-b33944c19332\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xrlvs" Oct 30 17:58:05.926499 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:05.926298 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc68d\" (UniqueName: \"kubernetes.io/projected/0242f980-e7c2-4799-964d-b33944c19332-kube-api-access-dc68d\") pod \"kserve-controller-manager-776cc66d47-xrlvs\" (UID: \"0242f980-e7c2-4799-964d-b33944c19332\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xrlvs" Oct 30 17:58:05.926499 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:05.926364 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e6cc7fd-dcda-490d-b3d0-661c28f95e1d-tls-certs\") pod \"model-serving-api-75bd89fdfc-6kc8h\" (UID: \"5e6cc7fd-dcda-490d-b3d0-661c28f95e1d\") " pod="opendatahub/model-serving-api-75bd89fdfc-6kc8h" Oct 30 17:58:05.928892 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:05.928863 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0242f980-e7c2-4799-964d-b33944c19332-cert\") pod \"kserve-controller-manager-776cc66d47-xrlvs\" (UID: \"0242f980-e7c2-4799-964d-b33944c19332\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xrlvs" Oct 30 17:58:05.928991 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:05.928902 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e6cc7fd-dcda-490d-b3d0-661c28f95e1d-tls-certs\") pod \"model-serving-api-75bd89fdfc-6kc8h\" (UID: \"5e6cc7fd-dcda-490d-b3d0-661c28f95e1d\") " pod="opendatahub/model-serving-api-75bd89fdfc-6kc8h" Oct 30 17:58:05.961028 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:05.960959 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc68d\" (UniqueName: \"kubernetes.io/projected/0242f980-e7c2-4799-964d-b33944c19332-kube-api-access-dc68d\") pod \"kserve-controller-manager-776cc66d47-xrlvs\" (UID: \"0242f980-e7c2-4799-964d-b33944c19332\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xrlvs" Oct 30 17:58:05.961154 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:05.961139 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vnn6\" (UniqueName: \"kubernetes.io/projected/5e6cc7fd-dcda-490d-b3d0-661c28f95e1d-kube-api-access-2vnn6\") pod \"model-serving-api-75bd89fdfc-6kc8h\" (UID: \"5e6cc7fd-dcda-490d-b3d0-661c28f95e1d\") " pod="opendatahub/model-serving-api-75bd89fdfc-6kc8h" Oct 30 17:58:06.008990 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:06.008957 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-xrlvs" Oct 30 17:58:06.088218 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:06.088178 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-6kc8h" Oct 30 17:58:06.310134 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:06.310011 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-xrlvs"] Oct 30 17:58:06.312885 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:58:06.312859 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0242f980_e7c2_4799_964d_b33944c19332.slice/crio-07b3ccb3f93153e2b003c658d2a6d45fa2a6b8c190103cfbd6263254a6ef5ba4 WatchSource:0}: Error finding container 07b3ccb3f93153e2b003c658d2a6d45fa2a6b8c190103cfbd6263254a6ef5ba4: Status 404 returned error can't find the container with id 07b3ccb3f93153e2b003c658d2a6d45fa2a6b8c190103cfbd6263254a6ef5ba4 Oct 30 17:58:06.399628 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:06.399594 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-6kc8h"] Oct 30 17:58:06.413965 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:58:06.413913 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e6cc7fd_dcda_490d_b3d0_661c28f95e1d.slice/crio-183007658e7a20a67cad3916109512ac3c9e1a137a96872409d9aabbe6f9f744 WatchSource:0}: Error finding container 183007658e7a20a67cad3916109512ac3c9e1a137a96872409d9aabbe6f9f744: Status 404 returned error can't find the container with id 183007658e7a20a67cad3916109512ac3c9e1a137a96872409d9aabbe6f9f744 Oct 30 17:58:06.426287 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:06.426252 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-xrlvs" event={"ID":"0242f980-e7c2-4799-964d-b33944c19332","Type":"ContainerStarted","Data":"07b3ccb3f93153e2b003c658d2a6d45fa2a6b8c190103cfbd6263254a6ef5ba4"} Oct 30 17:58:06.427792 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:06.427763 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-6kc8h" event={"ID":"5e6cc7fd-dcda-490d-b3d0-661c28f95e1d","Type":"ContainerStarted","Data":"183007658e7a20a67cad3916109512ac3c9e1a137a96872409d9aabbe6f9f744"} Oct 30 17:58:08.444230 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:08.444152 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-6kc8h" event={"ID":"5e6cc7fd-dcda-490d-b3d0-661c28f95e1d","Type":"ContainerStarted","Data":"b9b30d3871115052992a2b77a9c68a2e4f4219dc5d75a153dc6dba985f2adfed"} Oct 30 17:58:08.444745 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:08.444302 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-6kc8h" Oct 30 17:58:08.486067 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:08.485976 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-6kc8h" podStartSLOduration=1.7809291250000001 podStartE2EDuration="3.485963123s" totalImagesPullingTime="1.705033998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:08.483932667 +0000 UTC m=+340.971569702" watchObservedRunningTime="2026-10-30 17:58:08.485963123 +0000 UTC m=+340.973600157" Oct 30 17:58:09.458245 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:09.458205 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-xrlvs" event={"ID":"0242f980-e7c2-4799-964d-b33944c19332","Type":"ContainerStarted","Data":"6c9e718476f311931d70858eb7ece7ab1bfa2bc80f82b9ddb05efb598dbbe46b"} Oct 30 17:58:09.458665 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:09.458507 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-xrlvs" Oct 30 17:58:09.498157 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:09.498107 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-xrlvs" podStartSLOduration=2.300301647 podStartE2EDuration="4.498093326s" totalImagesPullingTime="2.197791679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:09.496183601 +0000 UTC m=+341.983820632" watchObservedRunningTime="2026-10-30 17:58:09.498093326 +0000 UTC m=+341.985730360" Oct 30 17:58:19.462298 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:19.462265 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-6kc8h" Oct 30 17:58:30.384226 ip-10-0-130-84 kubenswrapper[4455]: E1030 17:58:30.384194 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 17:58:30.965033 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:30.964996 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d8bcd8757-pjm4q"] Oct 30 17:58:30.988660 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:30.988620 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d8bcd8757-pjm4q" Oct 30 17:58:30.993246 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:30.993219 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d8bcd8757-pjm4q"] Oct 30 17:58:31.002949 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:31.002924 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 17:58:31.003962 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:31.003941 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8pmq2\"" Oct 30 17:58:31.014301 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:31.014284 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 17:58:31.157229 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:31.157195 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0212d69b-957b-4dd5-9b42-cc3fd6caa207-metrics-certs\") pod \"maas-api-6d8bcd8757-pjm4q\" (UID: \"0212d69b-957b-4dd5-9b42-cc3fd6caa207\") " pod="odh-ai-gateway-infra/maas-api-6d8bcd8757-pjm4q" Oct 30 17:58:31.157229 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:31.157231 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78cxp\" (UniqueName: \"kubernetes.io/projected/0212d69b-957b-4dd5-9b42-cc3fd6caa207-kube-api-access-78cxp\") pod \"maas-api-6d8bcd8757-pjm4q\" (UID: \"0212d69b-957b-4dd5-9b42-cc3fd6caa207\") " pod="odh-ai-gateway-infra/maas-api-6d8bcd8757-pjm4q" Oct 30 17:58:31.157475 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:31.157313 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0212d69b-957b-4dd5-9b42-cc3fd6caa207-maas-api-tls\") pod \"maas-api-6d8bcd8757-pjm4q\" (UID: \"0212d69b-957b-4dd5-9b42-cc3fd6caa207\") " pod="odh-ai-gateway-infra/maas-api-6d8bcd8757-pjm4q" Oct 30 17:58:31.157518 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:31.157498 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0212d69b-957b-4dd5-9b42-cc3fd6caa207-openshift-service-ca\") pod \"maas-api-6d8bcd8757-pjm4q\" (UID: \"0212d69b-957b-4dd5-9b42-cc3fd6caa207\") " pod="odh-ai-gateway-infra/maas-api-6d8bcd8757-pjm4q" Oct 30 17:58:31.258613 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:31.258581 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0212d69b-957b-4dd5-9b42-cc3fd6caa207-maas-api-tls\") pod \"maas-api-6d8bcd8757-pjm4q\" (UID: \"0212d69b-957b-4dd5-9b42-cc3fd6caa207\") " pod="odh-ai-gateway-infra/maas-api-6d8bcd8757-pjm4q" Oct 30 17:58:31.258775 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:31.258692 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0212d69b-957b-4dd5-9b42-cc3fd6caa207-openshift-service-ca\") pod \"maas-api-6d8bcd8757-pjm4q\" (UID: \"0212d69b-957b-4dd5-9b42-cc3fd6caa207\") " pod="odh-ai-gateway-infra/maas-api-6d8bcd8757-pjm4q" Oct 30 17:58:31.258775 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:31.258749 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0212d69b-957b-4dd5-9b42-cc3fd6caa207-metrics-certs\") pod \"maas-api-6d8bcd8757-pjm4q\" (UID: \"0212d69b-957b-4dd5-9b42-cc3fd6caa207\") " pod="odh-ai-gateway-infra/maas-api-6d8bcd8757-pjm4q" Oct 30 17:58:31.258847 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:31.258776 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78cxp\" (UniqueName: \"kubernetes.io/projected/0212d69b-957b-4dd5-9b42-cc3fd6caa207-kube-api-access-78cxp\") pod \"maas-api-6d8bcd8757-pjm4q\" (UID: \"0212d69b-957b-4dd5-9b42-cc3fd6caa207\") " pod="odh-ai-gateway-infra/maas-api-6d8bcd8757-pjm4q" Oct 30 17:58:31.259556 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:31.259488 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0212d69b-957b-4dd5-9b42-cc3fd6caa207-openshift-service-ca\") pod \"maas-api-6d8bcd8757-pjm4q\" (UID: \"0212d69b-957b-4dd5-9b42-cc3fd6caa207\") " pod="odh-ai-gateway-infra/maas-api-6d8bcd8757-pjm4q" Oct 30 17:58:31.261297 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:31.261277 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0212d69b-957b-4dd5-9b42-cc3fd6caa207-maas-api-tls\") pod \"maas-api-6d8bcd8757-pjm4q\" (UID: \"0212d69b-957b-4dd5-9b42-cc3fd6caa207\") " pod="odh-ai-gateway-infra/maas-api-6d8bcd8757-pjm4q" Oct 30 17:58:31.261441 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:31.261421 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0212d69b-957b-4dd5-9b42-cc3fd6caa207-metrics-certs\") pod \"maas-api-6d8bcd8757-pjm4q\" (UID: \"0212d69b-957b-4dd5-9b42-cc3fd6caa207\") " pod="odh-ai-gateway-infra/maas-api-6d8bcd8757-pjm4q" Oct 30 17:58:31.281573 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:31.281546 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78cxp\" (UniqueName: \"kubernetes.io/projected/0212d69b-957b-4dd5-9b42-cc3fd6caa207-kube-api-access-78cxp\") pod \"maas-api-6d8bcd8757-pjm4q\" (UID: \"0212d69b-957b-4dd5-9b42-cc3fd6caa207\") " pod="odh-ai-gateway-infra/maas-api-6d8bcd8757-pjm4q" Oct 30 17:58:31.304555 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:31.304528 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d8bcd8757-pjm4q" Oct 30 17:58:31.599689 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:31.599612 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d8bcd8757-pjm4q"] Oct 30 17:58:31.602084 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:58:31.602057 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0212d69b_957b_4dd5_9b42_cc3fd6caa207.slice/crio-ecb5c824f8701d173b04765727ccd65a9af09bf4fa7e4a947d2206c6df9a5f0f WatchSource:0}: Error finding container ecb5c824f8701d173b04765727ccd65a9af09bf4fa7e4a947d2206c6df9a5f0f: Status 404 returned error can't find the container with id ecb5c824f8701d173b04765727ccd65a9af09bf4fa7e4a947d2206c6df9a5f0f Oct 30 17:58:32.586643 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:32.586573 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d8bcd8757-pjm4q" event={"ID":"0212d69b-957b-4dd5-9b42-cc3fd6caa207","Type":"ContainerStarted","Data":"ecb5c824f8701d173b04765727ccd65a9af09bf4fa7e4a947d2206c6df9a5f0f"} Oct 30 17:58:34.601514 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:34.601479 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d8bcd8757-pjm4q" event={"ID":"0212d69b-957b-4dd5-9b42-cc3fd6caa207","Type":"ContainerStarted","Data":"9777ea7159c279644d13bf2756a93624bd0cadbb7b62fcde79c62aa0455c78a3"} Oct 30 17:58:34.601919 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:34.601570 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d8bcd8757-pjm4q" Oct 30 17:58:34.642016 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:34.641968 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d8bcd8757-pjm4q" podStartSLOduration=2.060934707 podStartE2EDuration="4.641954397s" totalImagesPullingTime="2.58101969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:34.641665725 +0000 UTC m=+367.129302759" watchObservedRunningTime="2026-10-30 17:58:34.641954397 +0000 UTC m=+367.129591430" Oct 30 17:58:40.470431 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:40.470395 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-xrlvs" Oct 30 17:58:40.612075 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:40.612042 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d8bcd8757-pjm4q" Oct 30 17:58:41.417554 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:41.417511 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7597569d74-sknct"] Oct 30 17:58:41.421228 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:41.421203 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7597569d74-sknct" Oct 30 17:58:41.435581 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:41.435555 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-fnvtk\"" Oct 30 17:58:41.449441 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:41.449405 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7597569d74-sknct"] Oct 30 17:58:41.454620 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:41.454585 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxrr7\" (UniqueName: \"kubernetes.io/projected/efa6e11c-a920-40df-b589-0611ec3b1a2f-kube-api-access-cxrr7\") pod \"authorino-7597569d74-sknct\" (UID: \"efa6e11c-a920-40df-b589-0611ec3b1a2f\") " pod="kuadrant-system/authorino-7597569d74-sknct" Oct 30 17:58:41.555764 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:41.555724 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxrr7\" (UniqueName: \"kubernetes.io/projected/efa6e11c-a920-40df-b589-0611ec3b1a2f-kube-api-access-cxrr7\") pod \"authorino-7597569d74-sknct\" (UID: \"efa6e11c-a920-40df-b589-0611ec3b1a2f\") " pod="kuadrant-system/authorino-7597569d74-sknct" Oct 30 17:58:41.580739 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:41.580707 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxrr7\" (UniqueName: \"kubernetes.io/projected/efa6e11c-a920-40df-b589-0611ec3b1a2f-kube-api-access-cxrr7\") pod \"authorino-7597569d74-sknct\" (UID: \"efa6e11c-a920-40df-b589-0611ec3b1a2f\") " pod="kuadrant-system/authorino-7597569d74-sknct" Oct 30 17:58:41.633356 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:41.633298 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7597569d74-sknct"] Oct 30 17:58:41.633553 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:41.633540 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7597569d74-sknct" Oct 30 17:58:41.658403 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:41.658186 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-65db9f99d7-gkmz2"] Oct 30 17:58:41.662673 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:41.662652 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65db9f99d7-gkmz2" Oct 30 17:58:41.681106 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:41.681043 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65db9f99d7-gkmz2"] Oct 30 17:58:41.714652 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:41.714626 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 17:58:41.759179 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:41.759142 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4d07bae5-b2f7-45cd-92f8-f3b4efaed2dd-tls-cert\") pod \"authorino-65db9f99d7-gkmz2\" (UID: \"4d07bae5-b2f7-45cd-92f8-f3b4efaed2dd\") " pod="kuadrant-system/authorino-65db9f99d7-gkmz2" Oct 30 17:58:41.759399 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:41.759376 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89gkt\" (UniqueName: \"kubernetes.io/projected/4d07bae5-b2f7-45cd-92f8-f3b4efaed2dd-kube-api-access-89gkt\") pod \"authorino-65db9f99d7-gkmz2\" (UID: \"4d07bae5-b2f7-45cd-92f8-f3b4efaed2dd\") " pod="kuadrant-system/authorino-65db9f99d7-gkmz2" Oct 30 17:58:41.766486 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:41.766461 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65db9f99d7-gkmz2"] Oct 30 17:58:41.766734 ip-10-0-130-84 kubenswrapper[4455]: E1030 17:58:41.766704 4455 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-89gkt tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-65db9f99d7-gkmz2" podUID="4d07bae5-b2f7-45cd-92f8-f3b4efaed2dd" Oct 30 17:58:41.860309 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:41.860275 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89gkt\" (UniqueName: \"kubernetes.io/projected/4d07bae5-b2f7-45cd-92f8-f3b4efaed2dd-kube-api-access-89gkt\") pod \"authorino-65db9f99d7-gkmz2\" (UID: \"4d07bae5-b2f7-45cd-92f8-f3b4efaed2dd\") " pod="kuadrant-system/authorino-65db9f99d7-gkmz2" Oct 30 17:58:41.860489 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:41.860368 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4d07bae5-b2f7-45cd-92f8-f3b4efaed2dd-tls-cert\") pod \"authorino-65db9f99d7-gkmz2\" (UID: \"4d07bae5-b2f7-45cd-92f8-f3b4efaed2dd\") " pod="kuadrant-system/authorino-65db9f99d7-gkmz2" Oct 30 17:58:41.863072 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:41.863050 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4d07bae5-b2f7-45cd-92f8-f3b4efaed2dd-tls-cert\") pod \"authorino-65db9f99d7-gkmz2\" (UID: \"4d07bae5-b2f7-45cd-92f8-f3b4efaed2dd\") " pod="kuadrant-system/authorino-65db9f99d7-gkmz2" Oct 30 17:58:41.882659 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:41.882633 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89gkt\" (UniqueName: \"kubernetes.io/projected/4d07bae5-b2f7-45cd-92f8-f3b4efaed2dd-kube-api-access-89gkt\") pod \"authorino-65db9f99d7-gkmz2\" (UID: \"4d07bae5-b2f7-45cd-92f8-f3b4efaed2dd\") " pod="kuadrant-system/authorino-65db9f99d7-gkmz2" Oct 30 17:58:41.940521 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:41.940344 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7597569d74-sknct"] Oct 30 17:58:41.943679 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:58:41.943646 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefa6e11c_a920_40df_b589_0611ec3b1a2f.slice/crio-e67478f05db0a9ea33e420fb29b89722ca7242f06fddd270863ea8117334b7c6 WatchSource:0}: Error finding container e67478f05db0a9ea33e420fb29b89722ca7242f06fddd270863ea8117334b7c6: Status 404 returned error can't find the container with id e67478f05db0a9ea33e420fb29b89722ca7242f06fddd270863ea8117334b7c6 Oct 30 17:58:42.643850 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:42.643808 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7597569d74-sknct" event={"ID":"efa6e11c-a920-40df-b589-0611ec3b1a2f","Type":"ContainerStarted","Data":"e67478f05db0a9ea33e420fb29b89722ca7242f06fddd270863ea8117334b7c6"} Oct 30 17:58:42.644377 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:42.643884 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65db9f99d7-gkmz2" Oct 30 17:58:42.655055 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:42.655033 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65db9f99d7-gkmz2" Oct 30 17:58:42.669484 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:42.669451 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d07bae5-b2f7-45cd-92f8-f3b4efaed2dd-kube-api-access-89gkt\" (UniqueName: \"kubernetes.io/projected/4d07bae5-b2f7-45cd-92f8-f3b4efaed2dd-kube-api-access-89gkt\") pod \"4d07bae5-b2f7-45cd-92f8-f3b4efaed2dd\" (UID: \"4d07bae5-b2f7-45cd-92f8-f3b4efaed2dd\") " Oct 30 17:58:42.669646 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:42.669578 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d07bae5-b2f7-45cd-92f8-f3b4efaed2dd-tls-cert\" (UniqueName: \"kubernetes.io/secret/4d07bae5-b2f7-45cd-92f8-f3b4efaed2dd-tls-cert\") pod \"4d07bae5-b2f7-45cd-92f8-f3b4efaed2dd\" (UID: \"4d07bae5-b2f7-45cd-92f8-f3b4efaed2dd\") " Oct 30 17:58:42.671826 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:42.671790 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d07bae5-b2f7-45cd-92f8-f3b4efaed2dd-kube-api-access-89gkt" pod "4d07bae5-b2f7-45cd-92f8-f3b4efaed2dd" (UID: "4d07bae5-b2f7-45cd-92f8-f3b4efaed2dd"). InnerVolumeSpecName "kube-api-access-89gkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:58:42.672019 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:42.672004 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d07bae5-b2f7-45cd-92f8-f3b4efaed2dd-tls-cert" pod "4d07bae5-b2f7-45cd-92f8-f3b4efaed2dd" (UID: "4d07bae5-b2f7-45cd-92f8-f3b4efaed2dd"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:58:42.771543 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:42.771508 4455 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4d07bae5-b2f7-45cd-92f8-f3b4efaed2dd-tls-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:58:42.771543 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:42.771544 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89gkt\" (UniqueName: \"kubernetes.io/projected/4d07bae5-b2f7-45cd-92f8-f3b4efaed2dd-kube-api-access-89gkt\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:58:43.243570 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:43.243533 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-55b8fd5845-fdl6z"] Oct 30 17:58:43.261872 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:43.261845 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55b8fd5845-fdl6z" Oct 30 17:58:43.273605 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:43.273575 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55b8fd5845-fdl6z"] Oct 30 17:58:43.276264 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:43.276239 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6e924469-24ea-4da5-9658-2c35efdea8dd-tls-cert\") pod \"authorino-55b8fd5845-fdl6z\" (UID: \"6e924469-24ea-4da5-9658-2c35efdea8dd\") " pod="kuadrant-system/authorino-55b8fd5845-fdl6z" Oct 30 17:58:43.276434 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:43.276393 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xzkp\" (UniqueName: \"kubernetes.io/projected/6e924469-24ea-4da5-9658-2c35efdea8dd-kube-api-access-6xzkp\") pod \"authorino-55b8fd5845-fdl6z\" (UID: \"6e924469-24ea-4da5-9658-2c35efdea8dd\") " pod="kuadrant-system/authorino-55b8fd5845-fdl6z" Oct 30 17:58:43.377076 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:43.377039 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xzkp\" (UniqueName: \"kubernetes.io/projected/6e924469-24ea-4da5-9658-2c35efdea8dd-kube-api-access-6xzkp\") pod \"authorino-55b8fd5845-fdl6z\" (UID: \"6e924469-24ea-4da5-9658-2c35efdea8dd\") " pod="kuadrant-system/authorino-55b8fd5845-fdl6z" Oct 30 17:58:43.378393 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:43.377170 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6e924469-24ea-4da5-9658-2c35efdea8dd-tls-cert\") pod \"authorino-55b8fd5845-fdl6z\" (UID: \"6e924469-24ea-4da5-9658-2c35efdea8dd\") " pod="kuadrant-system/authorino-55b8fd5845-fdl6z" Oct 30 17:58:43.379865 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:43.379835 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6e924469-24ea-4da5-9658-2c35efdea8dd-tls-cert\") pod \"authorino-55b8fd5845-fdl6z\" (UID: \"6e924469-24ea-4da5-9658-2c35efdea8dd\") " pod="kuadrant-system/authorino-55b8fd5845-fdl6z" Oct 30 17:58:43.399408 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:43.399384 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xzkp\" (UniqueName: \"kubernetes.io/projected/6e924469-24ea-4da5-9658-2c35efdea8dd-kube-api-access-6xzkp\") pod \"authorino-55b8fd5845-fdl6z\" (UID: \"6e924469-24ea-4da5-9658-2c35efdea8dd\") " pod="kuadrant-system/authorino-55b8fd5845-fdl6z" Oct 30 17:58:43.577149 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:43.577030 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55b8fd5845-fdl6z" Oct 30 17:58:43.653226 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:43.653184 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7597569d74-sknct" podUID="efa6e11c-a920-40df-b589-0611ec3b1a2f" containerName="authorino" containerID="cri-o://3b0402fb7630faed4ca0aa496e117ae7267aa0bc751eddb60211b2318e41141a" gracePeriod=30 Oct 30 17:58:43.653569 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:43.653304 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7597569d74-sknct" event={"ID":"efa6e11c-a920-40df-b589-0611ec3b1a2f","Type":"ContainerStarted","Data":"3b0402fb7630faed4ca0aa496e117ae7267aa0bc751eddb60211b2318e41141a"} Oct 30 17:58:43.656485 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:43.656260 4455 generic.go:356] "Generic (PLEG): container finished" podID="c0d316f7-79bb-4e92-be1e-306552cc08b6" containerID="8e3cc89db8afac75210239ce9725f99727895c2d2d2eaae4ccd1c57fad5b4294" exitCode=0 Oct 30 17:58:43.656485 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:43.656363 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" event={"ID":"c0d316f7-79bb-4e92-be1e-306552cc08b6","Type":"ContainerDied","Data":"8e3cc89db8afac75210239ce9725f99727895c2d2d2eaae4ccd1c57fad5b4294"} Oct 30 17:58:43.656485 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:43.656440 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65db9f99d7-gkmz2" Oct 30 17:58:43.658104 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:43.658082 4455 scope.go:122] "RemoveContainer" containerID="8e3cc89db8afac75210239ce9725f99727895c2d2d2eaae4ccd1c57fad5b4294" Oct 30 17:58:43.659496 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:43.659456 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 17:58:43.659577 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:43.659553 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 17:58:43.659655 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:43.659636 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 17:58:43.697035 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:43.696944 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7597569d74-sknct" podStartSLOduration=1.9253806980000001 podStartE2EDuration="2.69692527s" totalImagesPullingTime="771.544572ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:43.695016718 +0000 UTC m=+376.182653751" watchObservedRunningTime="2026-10-30 17:58:43.69692527 +0000 UTC m=+376.184562308" Oct 30 17:58:43.755841 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:43.755806 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65db9f99d7-gkmz2"] Oct 30 17:58:43.762027 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:43.761994 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-65db9f99d7-gkmz2"] Oct 30 17:58:43.886343 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:43.883802 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55b8fd5845-fdl6z"] Oct 30 17:58:43.897393 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:58:43.895784 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e924469_24ea_4da5_9658_2c35efdea8dd.slice/crio-cd2758db391d64295e0b3010293afb4f0ce10ee4d5f4cc9de64eeba843540314 WatchSource:0}: Error finding container cd2758db391d64295e0b3010293afb4f0ce10ee4d5f4cc9de64eeba843540314: Status 404 returned error can't find the container with id cd2758db391d64295e0b3010293afb4f0ce10ee4d5f4cc9de64eeba843540314 Oct 30 17:58:43.928678 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:43.928653 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7597569d74-sknct" Oct 30 17:58:43.990352 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:43.986745 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efa6e11c-a920-40df-b589-0611ec3b1a2f-kube-api-access-cxrr7\" (UniqueName: \"kubernetes.io/projected/efa6e11c-a920-40df-b589-0611ec3b1a2f-kube-api-access-cxrr7\") pod \"efa6e11c-a920-40df-b589-0611ec3b1a2f\" (UID: \"efa6e11c-a920-40df-b589-0611ec3b1a2f\") " Oct 30 17:58:44.005582 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:44.005524 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efa6e11c-a920-40df-b589-0611ec3b1a2f-kube-api-access-cxrr7" pod "efa6e11c-a920-40df-b589-0611ec3b1a2f" (UID: "efa6e11c-a920-40df-b589-0611ec3b1a2f"). InnerVolumeSpecName "kube-api-access-cxrr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:58:44.087799 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:44.087767 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxrr7\" (UniqueName: \"kubernetes.io/projected/efa6e11c-a920-40df-b589-0611ec3b1a2f-kube-api-access-cxrr7\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:58:44.685351 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:44.665673 4455 generic.go:356] "Generic (PLEG): container finished" podID="efa6e11c-a920-40df-b589-0611ec3b1a2f" containerID="3b0402fb7630faed4ca0aa496e117ae7267aa0bc751eddb60211b2318e41141a" exitCode=0 Oct 30 17:58:44.685351 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:44.665833 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7597569d74-sknct" Oct 30 17:58:44.685351 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:44.666268 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7597569d74-sknct" event={"ID":"efa6e11c-a920-40df-b589-0611ec3b1a2f","Type":"ContainerDied","Data":"3b0402fb7630faed4ca0aa496e117ae7267aa0bc751eddb60211b2318e41141a"} Oct 30 17:58:44.685351 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:44.666351 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7597569d74-sknct" event={"ID":"efa6e11c-a920-40df-b589-0611ec3b1a2f","Type":"ContainerDied","Data":"e67478f05db0a9ea33e420fb29b89722ca7242f06fddd270863ea8117334b7c6"} Oct 30 17:58:44.685351 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:44.666378 4455 scope.go:122] "RemoveContainer" containerID="3b0402fb7630faed4ca0aa496e117ae7267aa0bc751eddb60211b2318e41141a" Oct 30 17:58:44.685351 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:44.679760 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" event={"ID":"c0d316f7-79bb-4e92-be1e-306552cc08b6","Type":"ContainerStarted","Data":"fdc0fa3b3ae4701d0e2de2b8d5636b1943a7d33c331ce8a2491303e03eefb3a1"} Oct 30 17:58:44.696715 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:44.694605 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55b8fd5845-fdl6z" event={"ID":"6e924469-24ea-4da5-9658-2c35efdea8dd","Type":"ContainerStarted","Data":"6c501214c9bbb3d54b536a2ec92649f9979f929ecd39a0c52f794d63da79b2d3"} Oct 30 17:58:44.696715 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:44.694657 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55b8fd5845-fdl6z" event={"ID":"6e924469-24ea-4da5-9658-2c35efdea8dd","Type":"ContainerStarted","Data":"cd2758db391d64295e0b3010293afb4f0ce10ee4d5f4cc9de64eeba843540314"} Oct 30 17:58:44.708110 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:44.708084 4455 scope.go:122] "RemoveContainer" containerID="3b0402fb7630faed4ca0aa496e117ae7267aa0bc751eddb60211b2318e41141a" Oct 30 17:58:44.708569 ip-10-0-130-84 kubenswrapper[4455]: E1030 17:58:44.708538 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b0402fb7630faed4ca0aa496e117ae7267aa0bc751eddb60211b2318e41141a\": container with ID starting with 3b0402fb7630faed4ca0aa496e117ae7267aa0bc751eddb60211b2318e41141a not found: ID does not exist" containerID="3b0402fb7630faed4ca0aa496e117ae7267aa0bc751eddb60211b2318e41141a" Oct 30 17:58:44.708685 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:44.708583 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b0402fb7630faed4ca0aa496e117ae7267aa0bc751eddb60211b2318e41141a"} err="failed to get container status \"3b0402fb7630faed4ca0aa496e117ae7267aa0bc751eddb60211b2318e41141a\": rpc error: code = NotFound desc = could not find container \"3b0402fb7630faed4ca0aa496e117ae7267aa0bc751eddb60211b2318e41141a\": container with ID starting with 3b0402fb7630faed4ca0aa496e117ae7267aa0bc751eddb60211b2318e41141a not found: ID does not exist" Oct 30 17:58:44.761972 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:44.761896 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7597569d74-sknct"] Oct 30 17:58:44.766007 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:44.765977 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7597569d74-sknct"] Oct 30 17:58:44.815519 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:44.815471 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-55b8fd5845-fdl6z" podStartSLOduration=1.5303398910000001 podStartE2EDuration="1.815455634s" totalImagesPullingTime="285.115743ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:44.813473467 +0000 UTC m=+377.301110501" watchObservedRunningTime="2026-10-30 17:58:44.815455634 +0000 UTC m=+377.303092667" Oct 30 17:58:45.140165 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:45.140131 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:58:45.141573 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:45.141536 4455 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-h2gf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body= Oct 30 17:58:45.141692 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:45.141607 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" podUID="c0d316f7-79bb-4e92-be1e-306552cc08b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" Oct 30 17:58:45.632117 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:45.632084 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d07bae5-b2f7-45cd-92f8-f3b4efaed2dd" path="/var/lib/kubelet/pods/4d07bae5-b2f7-45cd-92f8-f3b4efaed2dd/volumes" Oct 30 17:58:45.632344 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:45.632311 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efa6e11c-a920-40df-b589-0611ec3b1a2f" path="/var/lib/kubelet/pods/efa6e11c-a920-40df-b589-0611ec3b1a2f/volumes" Oct 30 17:58:45.745961 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:45.745929 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:58:46.140962 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:46.140922 4455 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-h2gf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body= Oct 30 17:58:46.141140 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:46.140981 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" podUID="c0d316f7-79bb-4e92-be1e-306552cc08b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" Oct 30 17:58:47.140233 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:47.140197 4455 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-h2gf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body= Oct 30 17:58:47.140753 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:47.140273 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" podUID="c0d316f7-79bb-4e92-be1e-306552cc08b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" Oct 30 17:58:48.140158 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:48.140119 4455 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-h2gf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body= Oct 30 17:58:48.140373 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:48.140174 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" podUID="c0d316f7-79bb-4e92-be1e-306552cc08b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" Oct 30 17:58:49.140683 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:49.140643 4455 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-h2gf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body= Oct 30 17:58:49.141086 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:49.140697 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" podUID="c0d316f7-79bb-4e92-be1e-306552cc08b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" Oct 30 17:58:50.798398 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:50.798356 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:58:50.799811 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:50.799790 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 17:58:50.992385 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:50.992350 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d8bcd8757-pjm4q"] Oct 30 17:58:50.992620 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:50.992593 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d8bcd8757-pjm4q" podUID="0212d69b-957b-4dd5-9b42-cc3fd6caa207" containerName="maas-api" containerID="cri-o://9777ea7159c279644d13bf2756a93624bd0cadbb7b62fcde79c62aa0455c78a3" gracePeriod=30 Oct 30 17:58:51.241681 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:51.241655 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d8bcd8757-pjm4q" Oct 30 17:58:51.262708 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:51.262679 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0212d69b-957b-4dd5-9b42-cc3fd6caa207-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0212d69b-957b-4dd5-9b42-cc3fd6caa207-metrics-certs\") pod \"0212d69b-957b-4dd5-9b42-cc3fd6caa207\" (UID: \"0212d69b-957b-4dd5-9b42-cc3fd6caa207\") " Oct 30 17:58:51.262865 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:51.262776 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0212d69b-957b-4dd5-9b42-cc3fd6caa207-kube-api-access-78cxp\" (UniqueName: \"kubernetes.io/projected/0212d69b-957b-4dd5-9b42-cc3fd6caa207-kube-api-access-78cxp\") pod \"0212d69b-957b-4dd5-9b42-cc3fd6caa207\" (UID: \"0212d69b-957b-4dd5-9b42-cc3fd6caa207\") " Oct 30 17:58:51.262865 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:51.262809 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0212d69b-957b-4dd5-9b42-cc3fd6caa207-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0212d69b-957b-4dd5-9b42-cc3fd6caa207-maas-api-tls\") pod \"0212d69b-957b-4dd5-9b42-cc3fd6caa207\" (UID: \"0212d69b-957b-4dd5-9b42-cc3fd6caa207\") " Oct 30 17:58:51.262865 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:51.262851 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0212d69b-957b-4dd5-9b42-cc3fd6caa207-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0212d69b-957b-4dd5-9b42-cc3fd6caa207-openshift-service-ca\") pod \"0212d69b-957b-4dd5-9b42-cc3fd6caa207\" (UID: \"0212d69b-957b-4dd5-9b42-cc3fd6caa207\") " Oct 30 17:58:51.263309 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:51.263277 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0212d69b-957b-4dd5-9b42-cc3fd6caa207-openshift-service-ca" pod "0212d69b-957b-4dd5-9b42-cc3fd6caa207" (UID: "0212d69b-957b-4dd5-9b42-cc3fd6caa207"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:58:51.265099 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:51.265073 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0212d69b-957b-4dd5-9b42-cc3fd6caa207-metrics-certs" pod "0212d69b-957b-4dd5-9b42-cc3fd6caa207" (UID: "0212d69b-957b-4dd5-9b42-cc3fd6caa207"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:58:51.265218 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:51.265073 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0212d69b-957b-4dd5-9b42-cc3fd6caa207-kube-api-access-78cxp" pod "0212d69b-957b-4dd5-9b42-cc3fd6caa207" (UID: "0212d69b-957b-4dd5-9b42-cc3fd6caa207"). InnerVolumeSpecName "kube-api-access-78cxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:58:51.265218 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:51.265156 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0212d69b-957b-4dd5-9b42-cc3fd6caa207-maas-api-tls" pod "0212d69b-957b-4dd5-9b42-cc3fd6caa207" (UID: "0212d69b-957b-4dd5-9b42-cc3fd6caa207"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:58:51.364524 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:51.364435 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78cxp\" (UniqueName: \"kubernetes.io/projected/0212d69b-957b-4dd5-9b42-cc3fd6caa207-kube-api-access-78cxp\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:58:51.364524 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:51.364466 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0212d69b-957b-4dd5-9b42-cc3fd6caa207-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:58:51.364524 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:51.364475 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0212d69b-957b-4dd5-9b42-cc3fd6caa207-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:58:51.364524 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:51.364484 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0212d69b-957b-4dd5-9b42-cc3fd6caa207-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 17:58:51.745942 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:51.745910 4455 generic.go:356] "Generic (PLEG): container finished" podID="0212d69b-957b-4dd5-9b42-cc3fd6caa207" containerID="9777ea7159c279644d13bf2756a93624bd0cadbb7b62fcde79c62aa0455c78a3" exitCode=0 Oct 30 17:58:51.746091 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:51.745955 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d8bcd8757-pjm4q" Oct 30 17:58:51.746091 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:51.745998 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d8bcd8757-pjm4q" event={"ID":"0212d69b-957b-4dd5-9b42-cc3fd6caa207","Type":"ContainerDied","Data":"9777ea7159c279644d13bf2756a93624bd0cadbb7b62fcde79c62aa0455c78a3"} Oct 30 17:58:51.746091 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:51.746049 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d8bcd8757-pjm4q" event={"ID":"0212d69b-957b-4dd5-9b42-cc3fd6caa207","Type":"ContainerDied","Data":"ecb5c824f8701d173b04765727ccd65a9af09bf4fa7e4a947d2206c6df9a5f0f"} Oct 30 17:58:51.746091 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:51.746081 4455 scope.go:122] "RemoveContainer" containerID="9777ea7159c279644d13bf2756a93624bd0cadbb7b62fcde79c62aa0455c78a3" Oct 30 17:58:51.755969 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:51.755947 4455 scope.go:122] "RemoveContainer" containerID="9777ea7159c279644d13bf2756a93624bd0cadbb7b62fcde79c62aa0455c78a3" Oct 30 17:58:51.756254 ip-10-0-130-84 kubenswrapper[4455]: E1030 17:58:51.756231 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9777ea7159c279644d13bf2756a93624bd0cadbb7b62fcde79c62aa0455c78a3\": container with ID starting with 9777ea7159c279644d13bf2756a93624bd0cadbb7b62fcde79c62aa0455c78a3 not found: ID does not exist" containerID="9777ea7159c279644d13bf2756a93624bd0cadbb7b62fcde79c62aa0455c78a3" Oct 30 17:58:51.756303 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:51.756264 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9777ea7159c279644d13bf2756a93624bd0cadbb7b62fcde79c62aa0455c78a3"} err="failed to get container status \"9777ea7159c279644d13bf2756a93624bd0cadbb7b62fcde79c62aa0455c78a3\": rpc error: code = NotFound desc = could not find container \"9777ea7159c279644d13bf2756a93624bd0cadbb7b62fcde79c62aa0455c78a3\": container with ID starting with 9777ea7159c279644d13bf2756a93624bd0cadbb7b62fcde79c62aa0455c78a3 not found: ID does not exist" Oct 30 17:58:51.795498 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:51.795474 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d8bcd8757-pjm4q"] Oct 30 17:58:51.806475 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:51.806448 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d8bcd8757-pjm4q"] Oct 30 17:58:53.632461 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:58:53.632427 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0212d69b-957b-4dd5-9b42-cc3fd6caa207" path="/var/lib/kubelet/pods/0212d69b-957b-4dd5-9b42-cc3fd6caa207/volumes" Oct 30 17:59:00.384694 ip-10-0-130-84 kubenswrapper[4455]: E1030 17:59:00.384659 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 17:59:27.679027 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:27.678999 4455 scope.go:122] "RemoveContainer" containerID="b5ddf634ff70725c2cf974adf92251dc198ec7abfdc2035f99af6c3eb3aacef3" Oct 30 17:59:27.687472 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:27.687451 4455 scope.go:122] "RemoveContainer" containerID="85c5870e529178dd1f9118cf962378547cd8309cce3ba28134ce1683e4deefd0" Oct 30 17:59:27.695921 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:27.695903 4455 scope.go:122] "RemoveContainer" containerID="a878b23685252e9b053dd3da352dd9720ec2b6d74f46c3c0b269c6bfe2983a9e" Oct 30 17:59:27.703623 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:27.703604 4455 scope.go:122] "RemoveContainer" containerID="31e12d734fb8492037f5324ec732b0d744dfb30ab8cb32f403349e0c20348570" Oct 30 17:59:27.711640 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:27.711615 4455 scope.go:122] "RemoveContainer" containerID="054f0fd64a06441ce70dbcc616c2ae53824dbaf4c08562e1e43ab1eda50af52b" Oct 30 17:59:27.719629 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:27.719606 4455 scope.go:122] "RemoveContainer" containerID="22a74892463a1b031960f163705346a1d23615ad435300c572b654a0692ac1df" Oct 30 17:59:27.726525 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:27.726508 4455 scope.go:122] "RemoveContainer" containerID="bf5d65c13d922359caef794353cd693c50ba97b4fa42d0c5c3e0e028ccaef72d" Oct 30 17:59:27.735146 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:27.735119 4455 scope.go:122] "RemoveContainer" containerID="0f8cb3eacd62f7f0094c805da392ec86d6ca3e32e3d2cc628c596eee1ccbab15" Oct 30 17:59:30.384582 ip-10-0-130-84 kubenswrapper[4455]: E1030 17:59:30.384542 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 17:59:38.110025 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.109992 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr"] Oct 30 17:59:38.110440 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.110425 4455 cpu_manager.go:414] "containerMap: removing container" podUID="0212d69b-957b-4dd5-9b42-cc3fd6caa207" containerName="maas-api" Oct 30 17:59:38.110498 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.110441 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0212d69b-957b-4dd5-9b42-cc3fd6caa207" containerName="maas-api" Oct 30 17:59:38.110498 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.110479 4455 cpu_manager.go:414] "containerMap: removing container" podUID="efa6e11c-a920-40df-b589-0611ec3b1a2f" containerName="authorino" Oct 30 17:59:38.110498 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.110485 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efa6e11c-a920-40df-b589-0611ec3b1a2f" containerName="authorino" Oct 30 17:59:38.110592 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.110551 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="efa6e11c-a920-40df-b589-0611ec3b1a2f" containerName="authorino" Oct 30 17:59:38.110592 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.110560 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="0212d69b-957b-4dd5-9b42-cc3fd6caa207" containerName="maas-api" Oct 30 17:59:38.115030 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.115011 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" Oct 30 17:59:38.145653 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.145625 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr"] Oct 30 17:59:38.200787 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.200747 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqrh2\" (UniqueName: \"kubernetes.io/projected/d427f823-1810-42b1-b753-1790622bca7f-kube-api-access-tqrh2\") pod \"maas-default-gateway-openshift-default-67bf986f99-nrbjr\" (UID: \"d427f823-1810-42b1-b753-1790622bca7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" Oct 30 17:59:38.200969 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.200794 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d427f823-1810-42b1-b753-1790622bca7f-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-nrbjr\" (UID: \"d427f823-1810-42b1-b753-1790622bca7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" Oct 30 17:59:38.200969 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.200864 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d427f823-1810-42b1-b753-1790622bca7f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-nrbjr\" (UID: \"d427f823-1810-42b1-b753-1790622bca7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" Oct 30 17:59:38.200969 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.200900 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d427f823-1810-42b1-b753-1790622bca7f-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-nrbjr\" (UID: \"d427f823-1810-42b1-b753-1790622bca7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" Oct 30 17:59:38.200969 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.200924 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d427f823-1810-42b1-b753-1790622bca7f-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-nrbjr\" (UID: \"d427f823-1810-42b1-b753-1790622bca7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" Oct 30 17:59:38.201144 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.200973 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d427f823-1810-42b1-b753-1790622bca7f-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-nrbjr\" (UID: \"d427f823-1810-42b1-b753-1790622bca7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" Oct 30 17:59:38.201144 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.201018 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d427f823-1810-42b1-b753-1790622bca7f-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-nrbjr\" (UID: \"d427f823-1810-42b1-b753-1790622bca7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" Oct 30 17:59:38.201144 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.201059 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d427f823-1810-42b1-b753-1790622bca7f-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-nrbjr\" (UID: \"d427f823-1810-42b1-b753-1790622bca7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" Oct 30 17:59:38.201144 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.201095 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d427f823-1810-42b1-b753-1790622bca7f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-nrbjr\" (UID: \"d427f823-1810-42b1-b753-1790622bca7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" Oct 30 17:59:38.301989 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.301947 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqrh2\" (UniqueName: \"kubernetes.io/projected/d427f823-1810-42b1-b753-1790622bca7f-kube-api-access-tqrh2\") pod \"maas-default-gateway-openshift-default-67bf986f99-nrbjr\" (UID: \"d427f823-1810-42b1-b753-1790622bca7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" Oct 30 17:59:38.302168 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.301996 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d427f823-1810-42b1-b753-1790622bca7f-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-nrbjr\" (UID: \"d427f823-1810-42b1-b753-1790622bca7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" Oct 30 17:59:38.302168 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.302029 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d427f823-1810-42b1-b753-1790622bca7f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-nrbjr\" (UID: \"d427f823-1810-42b1-b753-1790622bca7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" Oct 30 17:59:38.302168 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.302056 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d427f823-1810-42b1-b753-1790622bca7f-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-nrbjr\" (UID: \"d427f823-1810-42b1-b753-1790622bca7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" Oct 30 17:59:38.302168 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.302090 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d427f823-1810-42b1-b753-1790622bca7f-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-nrbjr\" (UID: \"d427f823-1810-42b1-b753-1790622bca7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" Oct 30 17:59:38.302168 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.302117 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d427f823-1810-42b1-b753-1790622bca7f-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-nrbjr\" (UID: \"d427f823-1810-42b1-b753-1790622bca7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" Oct 30 17:59:38.302168 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.302161 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d427f823-1810-42b1-b753-1790622bca7f-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-nrbjr\" (UID: \"d427f823-1810-42b1-b753-1790622bca7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" Oct 30 17:59:38.302586 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.302207 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d427f823-1810-42b1-b753-1790622bca7f-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-nrbjr\" (UID: \"d427f823-1810-42b1-b753-1790622bca7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" Oct 30 17:59:38.302586 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.302263 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d427f823-1810-42b1-b753-1790622bca7f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-nrbjr\" (UID: \"d427f823-1810-42b1-b753-1790622bca7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" Oct 30 17:59:38.302586 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.302492 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d427f823-1810-42b1-b753-1790622bca7f-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-nrbjr\" (UID: \"d427f823-1810-42b1-b753-1790622bca7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" Oct 30 17:59:38.302586 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.302574 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d427f823-1810-42b1-b753-1790622bca7f-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-nrbjr\" (UID: \"d427f823-1810-42b1-b753-1790622bca7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" Oct 30 17:59:38.302758 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.302611 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d427f823-1810-42b1-b753-1790622bca7f-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-nrbjr\" (UID: \"d427f823-1810-42b1-b753-1790622bca7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" Oct 30 17:59:38.302758 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.302713 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d427f823-1810-42b1-b753-1790622bca7f-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-nrbjr\" (UID: \"d427f823-1810-42b1-b753-1790622bca7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" Oct 30 17:59:38.303094 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.303039 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d427f823-1810-42b1-b753-1790622bca7f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-nrbjr\" (UID: \"d427f823-1810-42b1-b753-1790622bca7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" Oct 30 17:59:38.304678 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.304656 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d427f823-1810-42b1-b753-1790622bca7f-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-nrbjr\" (UID: \"d427f823-1810-42b1-b753-1790622bca7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" Oct 30 17:59:38.304764 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.304707 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d427f823-1810-42b1-b753-1790622bca7f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-nrbjr\" (UID: \"d427f823-1810-42b1-b753-1790622bca7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" Oct 30 17:59:38.324890 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.324859 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d427f823-1810-42b1-b753-1790622bca7f-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-nrbjr\" (UID: \"d427f823-1810-42b1-b753-1790622bca7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" Oct 30 17:59:38.325106 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.325089 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqrh2\" (UniqueName: \"kubernetes.io/projected/d427f823-1810-42b1-b753-1790622bca7f-kube-api-access-tqrh2\") pod \"maas-default-gateway-openshift-default-67bf986f99-nrbjr\" (UID: \"d427f823-1810-42b1-b753-1790622bca7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" Oct 30 17:59:38.427584 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.427505 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" Oct 30 17:59:38.749181 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.748908 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr"] Oct 30 17:59:38.751298 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:59:38.751272 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd427f823_1810_42b1_b753_1790622bca7f.slice/crio-3eeb45094afffa96525158d42ec6075320b2fdfaa10ff85b54500083ca629e44 WatchSource:0}: Error finding container 3eeb45094afffa96525158d42ec6075320b2fdfaa10ff85b54500083ca629e44: Status 404 returned error can't find the container with id 3eeb45094afffa96525158d42ec6075320b2fdfaa10ff85b54500083ca629e44 Oct 30 17:59:38.753453 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.753421 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 17:59:38.753538 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.753487 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 17:59:38.753538 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.753521 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 17:59:38.995560 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.995506 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" event={"ID":"d427f823-1810-42b1-b753-1790622bca7f","Type":"ContainerStarted","Data":"5f927012528e93de20539705d6bcc0cd3032ff215d6567801a81f72527a6d497"} Oct 30 17:59:38.995746 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:38.995581 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" event={"ID":"d427f823-1810-42b1-b753-1790622bca7f","Type":"ContainerStarted","Data":"3eeb45094afffa96525158d42ec6075320b2fdfaa10ff85b54500083ca629e44"} Oct 30 17:59:39.042656 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:39.042554 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" podStartSLOduration=1.042539104 podStartE2EDuration="1.042539104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:59:39.039716893 +0000 UTC m=+431.527353927" watchObservedRunningTime="2026-10-30 17:59:39.042539104 +0000 UTC m=+431.530176138" Oct 30 17:59:39.428239 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:39.428154 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" Oct 30 17:59:39.429342 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:39.429300 4455 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-nrbjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 17:59:39.429403 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:39.429372 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" podUID="d427f823-1810-42b1-b753-1790622bca7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 17:59:40.196670 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:40.196629 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-8pcnj"] Oct 30 17:59:40.199558 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:40.199534 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-8pcnj" Oct 30 17:59:40.222492 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:40.222464 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/febf052c-1f46-4866-a1a3-88499118ae28-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-8pcnj\" (UID: \"febf052c-1f46-4866-a1a3-88499118ae28\") " pod="opendatahub/ai-gateway-controller-7d85675f98-8pcnj" Oct 30 17:59:40.222642 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:40.222574 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76sxk\" (UniqueName: \"kubernetes.io/projected/febf052c-1f46-4866-a1a3-88499118ae28-kube-api-access-76sxk\") pod \"ai-gateway-controller-7d85675f98-8pcnj\" (UID: \"febf052c-1f46-4866-a1a3-88499118ae28\") " pod="opendatahub/ai-gateway-controller-7d85675f98-8pcnj" Oct 30 17:59:40.228402 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:40.228374 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 17:59:40.228569 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:40.228432 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 17:59:40.228569 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:40.228453 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-xpnrl\"" Oct 30 17:59:40.231823 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:40.231795 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-8pcnj"] Oct 30 17:59:40.324413 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:40.324359 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/febf052c-1f46-4866-a1a3-88499118ae28-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-8pcnj\" (UID: \"febf052c-1f46-4866-a1a3-88499118ae28\") " pod="opendatahub/ai-gateway-controller-7d85675f98-8pcnj" Oct 30 17:59:40.325086 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:40.324516 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76sxk\" (UniqueName: \"kubernetes.io/projected/febf052c-1f46-4866-a1a3-88499118ae28-kube-api-access-76sxk\") pod \"ai-gateway-controller-7d85675f98-8pcnj\" (UID: \"febf052c-1f46-4866-a1a3-88499118ae28\") " pod="opendatahub/ai-gateway-controller-7d85675f98-8pcnj" Oct 30 17:59:40.327390 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:40.327367 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/febf052c-1f46-4866-a1a3-88499118ae28-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-8pcnj\" (UID: \"febf052c-1f46-4866-a1a3-88499118ae28\") " pod="opendatahub/ai-gateway-controller-7d85675f98-8pcnj" Oct 30 17:59:40.347803 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:40.347772 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76sxk\" (UniqueName: \"kubernetes.io/projected/febf052c-1f46-4866-a1a3-88499118ae28-kube-api-access-76sxk\") pod \"ai-gateway-controller-7d85675f98-8pcnj\" (UID: \"febf052c-1f46-4866-a1a3-88499118ae28\") " pod="opendatahub/ai-gateway-controller-7d85675f98-8pcnj" Oct 30 17:59:40.428872 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:40.428835 4455 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-nrbjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 17:59:40.429262 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:40.428891 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" podUID="d427f823-1810-42b1-b753-1790622bca7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 17:59:40.512835 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:40.512798 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-8pcnj" Oct 30 17:59:40.827603 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:40.827570 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-8pcnj"] Oct 30 17:59:41.006122 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:41.006079 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-8pcnj" event={"ID":"febf052c-1f46-4866-a1a3-88499118ae28","Type":"ContainerStarted","Data":"acebe1e834400c219ee78fe378d9095ff357487f5ec2e3aa2abc05753687c769"} Oct 30 17:59:41.428538 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:41.428494 4455 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-nrbjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 17:59:41.428721 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:41.428578 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" podUID="d427f823-1810-42b1-b753-1790622bca7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 17:59:42.428215 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:42.428143 4455 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-nrbjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 17:59:42.428215 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:42.428205 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" podUID="d427f823-1810-42b1-b753-1790622bca7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 17:59:43.018984 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:43.018947 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-8pcnj" event={"ID":"febf052c-1f46-4866-a1a3-88499118ae28","Type":"ContainerStarted","Data":"7c9a2e2db123926446cd83296df4a9aaf4074c433e0e0e45e7d23391a0f98047"} Oct 30 17:59:43.019241 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:43.019083 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-8pcnj" Oct 30 17:59:43.063651 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:43.063595 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-8pcnj" podStartSLOduration=1.271985419 podStartE2EDuration="3.063580861s" totalImagesPullingTime="1.791595442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:59:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:59:43.059765963 +0000 UTC m=+435.547402996" watchObservedRunningTime="2026-10-30 17:59:43.063580861 +0000 UTC m=+435.551217895" Oct 30 17:59:43.428464 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:43.428377 4455 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-nrbjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 17:59:43.428464 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:43.428431 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" podUID="d427f823-1810-42b1-b753-1790622bca7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 17:59:44.130850 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:44.130814 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-znczm"] Oct 30 17:59:44.133664 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:44.133645 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-znczm" Oct 30 17:59:44.147900 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:44.147873 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-7zw2s\"" Oct 30 17:59:44.148022 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:44.148011 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 17:59:44.161970 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:44.161938 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-znczm"] Oct 30 17:59:44.209739 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:44.209710 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-9l4x8"] Oct 30 17:59:44.214620 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:44.214589 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-9l4x8" Oct 30 17:59:44.239272 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:44.239239 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-9l4x8"] Oct 30 17:59:44.291351 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:44.290863 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hrtv\" (UniqueName: \"kubernetes.io/projected/73b20a69-5907-4bec-a4c6-6b31b46231c4-kube-api-access-5hrtv\") pod \"payload-pre-processing-7b98bb678c-znczm\" (UID: \"73b20a69-5907-4bec-a4c6-6b31b46231c4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-znczm" Oct 30 17:59:44.291351 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:44.290931 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73b20a69-5907-4bec-a4c6-6b31b46231c4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-znczm\" (UID: \"73b20a69-5907-4bec-a4c6-6b31b46231c4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-znczm" Oct 30 17:59:44.291351 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:44.290982 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbvjb\" (UniqueName: \"kubernetes.io/projected/909b1ff6-6d4a-4887-9b59-efe2806ac0e8-kube-api-access-pbvjb\") pod \"payload-processing-5f78687d5b-9l4x8\" (UID: \"909b1ff6-6d4a-4887-9b59-efe2806ac0e8\") " pod="openshift-ingress/payload-processing-5f78687d5b-9l4x8" Oct 30 17:59:44.291351 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:44.291068 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/909b1ff6-6d4a-4887-9b59-efe2806ac0e8-plugins-config-volume\") pod \"payload-processing-5f78687d5b-9l4x8\" (UID: \"909b1ff6-6d4a-4887-9b59-efe2806ac0e8\") " pod="openshift-ingress/payload-processing-5f78687d5b-9l4x8" Oct 30 17:59:44.393398 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:44.391842 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/909b1ff6-6d4a-4887-9b59-efe2806ac0e8-plugins-config-volume\") pod \"payload-processing-5f78687d5b-9l4x8\" (UID: \"909b1ff6-6d4a-4887-9b59-efe2806ac0e8\") " pod="openshift-ingress/payload-processing-5f78687d5b-9l4x8" Oct 30 17:59:44.393398 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:44.391960 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hrtv\" (UniqueName: \"kubernetes.io/projected/73b20a69-5907-4bec-a4c6-6b31b46231c4-kube-api-access-5hrtv\") pod \"payload-pre-processing-7b98bb678c-znczm\" (UID: \"73b20a69-5907-4bec-a4c6-6b31b46231c4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-znczm" Oct 30 17:59:44.393398 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:44.392006 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73b20a69-5907-4bec-a4c6-6b31b46231c4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-znczm\" (UID: \"73b20a69-5907-4bec-a4c6-6b31b46231c4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-znczm" Oct 30 17:59:44.393398 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:44.392054 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbvjb\" (UniqueName: \"kubernetes.io/projected/909b1ff6-6d4a-4887-9b59-efe2806ac0e8-kube-api-access-pbvjb\") pod \"payload-processing-5f78687d5b-9l4x8\" (UID: \"909b1ff6-6d4a-4887-9b59-efe2806ac0e8\") " pod="openshift-ingress/payload-processing-5f78687d5b-9l4x8" Oct 30 17:59:44.393727 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:44.393420 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73b20a69-5907-4bec-a4c6-6b31b46231c4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-znczm\" (UID: \"73b20a69-5907-4bec-a4c6-6b31b46231c4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-znczm" Oct 30 17:59:44.393727 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:44.393443 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/909b1ff6-6d4a-4887-9b59-efe2806ac0e8-plugins-config-volume\") pod \"payload-processing-5f78687d5b-9l4x8\" (UID: \"909b1ff6-6d4a-4887-9b59-efe2806ac0e8\") " pod="openshift-ingress/payload-processing-5f78687d5b-9l4x8" Oct 30 17:59:44.415175 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:44.415144 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbvjb\" (UniqueName: \"kubernetes.io/projected/909b1ff6-6d4a-4887-9b59-efe2806ac0e8-kube-api-access-pbvjb\") pod \"payload-processing-5f78687d5b-9l4x8\" (UID: \"909b1ff6-6d4a-4887-9b59-efe2806ac0e8\") " pod="openshift-ingress/payload-processing-5f78687d5b-9l4x8" Oct 30 17:59:44.415341 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:44.415146 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hrtv\" (UniqueName: \"kubernetes.io/projected/73b20a69-5907-4bec-a4c6-6b31b46231c4-kube-api-access-5hrtv\") pod \"payload-pre-processing-7b98bb678c-znczm\" (UID: \"73b20a69-5907-4bec-a4c6-6b31b46231c4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-znczm" Oct 30 17:59:44.428787 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:44.428760 4455 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-nrbjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 17:59:44.429124 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:44.428807 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" podUID="d427f823-1810-42b1-b753-1790622bca7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 17:59:44.445605 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:44.445580 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-znczm" Oct 30 17:59:44.530760 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:44.530730 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-9l4x8" Oct 30 17:59:44.767458 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:44.767428 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-znczm"] Oct 30 17:59:44.769485 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:59:44.769453 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b20a69_5907_4bec_a4c6_6b31b46231c4.slice/crio-0ddcab79c353b5cf4842a7e4cd9c31f7cb370235038847429e6a7d2d9aa4f0f9 WatchSource:0}: Error finding container 0ddcab79c353b5cf4842a7e4cd9c31f7cb370235038847429e6a7d2d9aa4f0f9: Status 404 returned error can't find the container with id 0ddcab79c353b5cf4842a7e4cd9c31f7cb370235038847429e6a7d2d9aa4f0f9 Oct 30 17:59:44.852006 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:44.851971 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-9l4x8"] Oct 30 17:59:44.855538 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:59:44.855506 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod909b1ff6_6d4a_4887_9b59_efe2806ac0e8.slice/crio-a202401c56047eefae7c96981b15b1e9e097f5bf488049f3c2d3f9009227d1d6 WatchSource:0}: Error finding container a202401c56047eefae7c96981b15b1e9e097f5bf488049f3c2d3f9009227d1d6: Status 404 returned error can't find the container with id a202401c56047eefae7c96981b15b1e9e097f5bf488049f3c2d3f9009227d1d6 Oct 30 17:59:45.032475 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:45.032360 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-9l4x8" event={"ID":"909b1ff6-6d4a-4887-9b59-efe2806ac0e8","Type":"ContainerStarted","Data":"a202401c56047eefae7c96981b15b1e9e097f5bf488049f3c2d3f9009227d1d6"} Oct 30 17:59:45.033965 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:45.033936 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-znczm" event={"ID":"73b20a69-5907-4bec-a4c6-6b31b46231c4","Type":"ContainerStarted","Data":"0ddcab79c353b5cf4842a7e4cd9c31f7cb370235038847429e6a7d2d9aa4f0f9"} Oct 30 17:59:45.460583 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:45.460488 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" Oct 30 17:59:45.461689 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:45.461664 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" Oct 30 17:59:45.462397 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:45.462338 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nrbjr" Oct 30 17:59:48.054743 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:48.054707 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-znczm" event={"ID":"73b20a69-5907-4bec-a4c6-6b31b46231c4","Type":"ContainerStarted","Data":"b07b5e2b124f7aba7b1dc5e502b7abd5dcb7cfe92e78144b3173b3fa744db4b5"} Oct 30 17:59:48.055183 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:48.054872 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-znczm" Oct 30 17:59:48.056271 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:48.056245 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-9l4x8" event={"ID":"909b1ff6-6d4a-4887-9b59-efe2806ac0e8","Type":"ContainerStarted","Data":"5ae932c84e3cb0c9f9e3c8718298269b03123f65bbef6be4783b04cf71a33e6b"} Oct 30 17:59:48.056385 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:48.056368 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-9l4x8" Oct 30 17:59:48.094560 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:48.094493 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-znczm" podStartSLOduration=1.8079558709999999 podStartE2EDuration="4.094479818s" totalImagesPullingTime="2.286523947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:59:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:59:48.093008863 +0000 UTC m=+440.580645898" watchObservedRunningTime="2026-10-30 17:59:48.094479818 +0000 UTC m=+440.582116852" Oct 30 17:59:48.134042 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:48.133985 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-9l4x8" podStartSLOduration=1.88686843 podStartE2EDuration="4.133968082s" totalImagesPullingTime="2.247099652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:59:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:59:48.131518705 +0000 UTC m=+440.619155739" watchObservedRunningTime="2026-10-30 17:59:48.133968082 +0000 UTC m=+440.621605115" Oct 30 17:59:54.027612 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:54.027582 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-8pcnj" Oct 30 17:59:54.323539 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:54.323451 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-cf84bc4f4-dmpmz"] Oct 30 17:59:54.326580 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:54.326557 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-cf84bc4f4-dmpmz" Oct 30 17:59:54.349111 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:54.349082 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-cf84bc4f4-dmpmz"] Oct 30 17:59:54.495171 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:54.495142 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d464k\" (UniqueName: \"kubernetes.io/projected/7442c249-6e2c-4bcf-9383-b1ab7355e02e-kube-api-access-d464k\") pod \"ai-gateway-controller-cf84bc4f4-dmpmz\" (UID: \"7442c249-6e2c-4bcf-9383-b1ab7355e02e\") " pod="opendatahub/ai-gateway-controller-cf84bc4f4-dmpmz" Oct 30 17:59:54.495400 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:54.495207 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7442c249-6e2c-4bcf-9383-b1ab7355e02e-metrics-certs\") pod \"ai-gateway-controller-cf84bc4f4-dmpmz\" (UID: \"7442c249-6e2c-4bcf-9383-b1ab7355e02e\") " pod="opendatahub/ai-gateway-controller-cf84bc4f4-dmpmz" Oct 30 17:59:54.596179 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:54.596080 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7442c249-6e2c-4bcf-9383-b1ab7355e02e-metrics-certs\") pod \"ai-gateway-controller-cf84bc4f4-dmpmz\" (UID: \"7442c249-6e2c-4bcf-9383-b1ab7355e02e\") " pod="opendatahub/ai-gateway-controller-cf84bc4f4-dmpmz" Oct 30 17:59:54.596402 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:54.596210 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d464k\" (UniqueName: \"kubernetes.io/projected/7442c249-6e2c-4bcf-9383-b1ab7355e02e-kube-api-access-d464k\") pod \"ai-gateway-controller-cf84bc4f4-dmpmz\" (UID: \"7442c249-6e2c-4bcf-9383-b1ab7355e02e\") " pod="opendatahub/ai-gateway-controller-cf84bc4f4-dmpmz" Oct 30 17:59:54.598819 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:54.598795 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7442c249-6e2c-4bcf-9383-b1ab7355e02e-metrics-certs\") pod \"ai-gateway-controller-cf84bc4f4-dmpmz\" (UID: \"7442c249-6e2c-4bcf-9383-b1ab7355e02e\") " pod="opendatahub/ai-gateway-controller-cf84bc4f4-dmpmz" Oct 30 17:59:54.620192 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:54.620155 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d464k\" (UniqueName: \"kubernetes.io/projected/7442c249-6e2c-4bcf-9383-b1ab7355e02e-kube-api-access-d464k\") pod \"ai-gateway-controller-cf84bc4f4-dmpmz\" (UID: \"7442c249-6e2c-4bcf-9383-b1ab7355e02e\") " pod="opendatahub/ai-gateway-controller-cf84bc4f4-dmpmz" Oct 30 17:59:54.639998 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:54.639966 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-cf84bc4f4-dmpmz" Oct 30 17:59:54.934986 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:54.934945 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-cf84bc4f4-dmpmz"] Oct 30 17:59:54.937011 ip-10-0-130-84 kubenswrapper[4455]: W1030 17:59:54.936979 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7442c249_6e2c_4bcf_9383_b1ab7355e02e.slice/crio-565138bd923c6e9a6eec3d7d06cc67d4581b81ca002b2c5be4543dc332ff57dc WatchSource:0}: Error finding container 565138bd923c6e9a6eec3d7d06cc67d4581b81ca002b2c5be4543dc332ff57dc: Status 404 returned error can't find the container with id 565138bd923c6e9a6eec3d7d06cc67d4581b81ca002b2c5be4543dc332ff57dc Oct 30 17:59:55.094332 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:55.094287 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-cf84bc4f4-dmpmz" event={"ID":"7442c249-6e2c-4bcf-9383-b1ab7355e02e","Type":"ContainerStarted","Data":"565138bd923c6e9a6eec3d7d06cc67d4581b81ca002b2c5be4543dc332ff57dc"} Oct 30 17:59:56.101185 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:56.101154 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-cf84bc4f4-dmpmz" event={"ID":"7442c249-6e2c-4bcf-9383-b1ab7355e02e","Type":"ContainerStarted","Data":"e688f08e3ebc21112f9a7f550d8cdcc41a83f60ce2e046185e5bf71c29eaebf1"} Oct 30 17:59:56.101615 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:56.101269 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-cf84bc4f4-dmpmz" Oct 30 17:59:56.142694 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:56.141825 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-cf84bc4f4-dmpmz" podStartSLOduration=1.831300224 podStartE2EDuration="2.141810634s" totalImagesPullingTime="310.51041ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:59:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:59:56.140577346 +0000 UTC m=+448.628214360" watchObservedRunningTime="2026-10-30 17:59:56.141810634 +0000 UTC m=+448.629447668" Oct 30 17:59:59.064830 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:59.064800 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-9l4x8" Oct 30 17:59:59.065308 ip-10-0-130-84 kubenswrapper[4455]: I1030 17:59:59.064969 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-znczm" Oct 30 18:00:00.160293 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:00.160257 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889720-snxhj"] Oct 30 18:00:00.164052 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:00.164035 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-snxhj" Oct 30 18:00:00.178547 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:00.178530 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:00:00.192169 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:00.192142 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889720-snxhj"] Oct 30 18:00:00.252124 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:00.252084 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtwg2\" (UniqueName: \"kubernetes.io/projected/d6fd4787-f2fc-4567-b223-8dcd356163cb-kube-api-access-vtwg2\") pod \"maas-api-key-cleanup-29889720-snxhj\" (UID: \"d6fd4787-f2fc-4567-b223-8dcd356163cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-snxhj" Oct 30 18:00:00.252298 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:00.252147 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6fd4787-f2fc-4567-b223-8dcd356163cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889720-snxhj\" (UID: \"d6fd4787-f2fc-4567-b223-8dcd356163cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-snxhj" Oct 30 18:00:00.353530 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:00.353494 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtwg2\" (UniqueName: \"kubernetes.io/projected/d6fd4787-f2fc-4567-b223-8dcd356163cb-kube-api-access-vtwg2\") pod \"maas-api-key-cleanup-29889720-snxhj\" (UID: \"d6fd4787-f2fc-4567-b223-8dcd356163cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-snxhj" Oct 30 18:00:00.353712 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:00.353595 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6fd4787-f2fc-4567-b223-8dcd356163cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889720-snxhj\" (UID: \"d6fd4787-f2fc-4567-b223-8dcd356163cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-snxhj" Oct 30 18:00:00.354210 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:00.354182 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6fd4787-f2fc-4567-b223-8dcd356163cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889720-snxhj\" (UID: \"d6fd4787-f2fc-4567-b223-8dcd356163cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-snxhj" Oct 30 18:00:00.377048 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:00.377010 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtwg2\" (UniqueName: \"kubernetes.io/projected/d6fd4787-f2fc-4567-b223-8dcd356163cb-kube-api-access-vtwg2\") pod \"maas-api-key-cleanup-29889720-snxhj\" (UID: \"d6fd4787-f2fc-4567-b223-8dcd356163cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-snxhj" Oct 30 18:00:00.385017 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:00:00.384976 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:00:00.477993 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:00.477959 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-snxhj" Oct 30 18:00:00.777667 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:00.777640 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889720-snxhj"] Oct 30 18:00:00.780165 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:00:00.780142 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6fd4787_f2fc_4567_b223_8dcd356163cb.slice/crio-92c2d740d4876e4907a26b2ad1863864f20ae64f16845747bcfa9706a2f9b4a6 WatchSource:0}: Error finding container 92c2d740d4876e4907a26b2ad1863864f20ae64f16845747bcfa9706a2f9b4a6: Status 404 returned error can't find the container with id 92c2d740d4876e4907a26b2ad1863864f20ae64f16845747bcfa9706a2f9b4a6 Oct 30 18:00:01.129253 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:01.129169 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-snxhj" event={"ID":"d6fd4787-f2fc-4567-b223-8dcd356163cb","Type":"ContainerStarted","Data":"92c2d740d4876e4907a26b2ad1863864f20ae64f16845747bcfa9706a2f9b4a6"} Oct 30 18:00:07.108691 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:07.108659 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-cf84bc4f4-dmpmz" Oct 30 18:00:07.182095 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:07.182061 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-8pcnj"] Oct 30 18:00:07.182298 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:07.182276 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-8pcnj" podUID="febf052c-1f46-4866-a1a3-88499118ae28" containerName="manager" containerID="cri-o://7c9a2e2db123926446cd83296df4a9aaf4074c433e0e0e45e7d23391a0f98047" gracePeriod=30 Oct 30 18:00:07.440885 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:07.440862 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-8pcnj" Oct 30 18:00:07.521464 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:07.521434 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/febf052c-1f46-4866-a1a3-88499118ae28-kube-api-access-76sxk\" (UniqueName: \"kubernetes.io/projected/febf052c-1f46-4866-a1a3-88499118ae28-kube-api-access-76sxk\") pod \"febf052c-1f46-4866-a1a3-88499118ae28\" (UID: \"febf052c-1f46-4866-a1a3-88499118ae28\") " Oct 30 18:00:07.521663 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:07.521600 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/febf052c-1f46-4866-a1a3-88499118ae28-metrics-certs\" (UniqueName: \"kubernetes.io/secret/febf052c-1f46-4866-a1a3-88499118ae28-metrics-certs\") pod \"febf052c-1f46-4866-a1a3-88499118ae28\" (UID: \"febf052c-1f46-4866-a1a3-88499118ae28\") " Oct 30 18:00:07.523799 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:07.523772 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/febf052c-1f46-4866-a1a3-88499118ae28-metrics-certs" pod "febf052c-1f46-4866-a1a3-88499118ae28" (UID: "febf052c-1f46-4866-a1a3-88499118ae28"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:00:07.523920 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:07.523794 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/febf052c-1f46-4866-a1a3-88499118ae28-kube-api-access-76sxk" pod "febf052c-1f46-4866-a1a3-88499118ae28" (UID: "febf052c-1f46-4866-a1a3-88499118ae28"). InnerVolumeSpecName "kube-api-access-76sxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:00:07.623274 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:07.623237 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/febf052c-1f46-4866-a1a3-88499118ae28-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:00:07.623274 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:07.623267 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76sxk\" (UniqueName: \"kubernetes.io/projected/febf052c-1f46-4866-a1a3-88499118ae28-kube-api-access-76sxk\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:00:08.166000 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:08.165957 4455 generic.go:356] "Generic (PLEG): container finished" podID="febf052c-1f46-4866-a1a3-88499118ae28" containerID="7c9a2e2db123926446cd83296df4a9aaf4074c433e0e0e45e7d23391a0f98047" exitCode=0 Oct 30 18:00:08.166446 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:08.166008 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-8pcnj" Oct 30 18:00:08.166446 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:08.166039 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-8pcnj" event={"ID":"febf052c-1f46-4866-a1a3-88499118ae28","Type":"ContainerDied","Data":"7c9a2e2db123926446cd83296df4a9aaf4074c433e0e0e45e7d23391a0f98047"} Oct 30 18:00:08.166446 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:08.166094 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-8pcnj" event={"ID":"febf052c-1f46-4866-a1a3-88499118ae28","Type":"ContainerDied","Data":"acebe1e834400c219ee78fe378d9095ff357487f5ec2e3aa2abc05753687c769"} Oct 30 18:00:08.166446 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:08.166115 4455 scope.go:122] "RemoveContainer" containerID="7c9a2e2db123926446cd83296df4a9aaf4074c433e0e0e45e7d23391a0f98047" Oct 30 18:00:08.175594 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:08.175576 4455 scope.go:122] "RemoveContainer" containerID="7c9a2e2db123926446cd83296df4a9aaf4074c433e0e0e45e7d23391a0f98047" Oct 30 18:00:08.175902 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:00:08.175876 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c9a2e2db123926446cd83296df4a9aaf4074c433e0e0e45e7d23391a0f98047\": container with ID starting with 7c9a2e2db123926446cd83296df4a9aaf4074c433e0e0e45e7d23391a0f98047 not found: ID does not exist" containerID="7c9a2e2db123926446cd83296df4a9aaf4074c433e0e0e45e7d23391a0f98047" Oct 30 18:00:08.175997 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:08.175906 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c9a2e2db123926446cd83296df4a9aaf4074c433e0e0e45e7d23391a0f98047"} err="failed to get container status \"7c9a2e2db123926446cd83296df4a9aaf4074c433e0e0e45e7d23391a0f98047\": rpc error: code = NotFound desc = could not find container \"7c9a2e2db123926446cd83296df4a9aaf4074c433e0e0e45e7d23391a0f98047\": container with ID starting with 7c9a2e2db123926446cd83296df4a9aaf4074c433e0e0e45e7d23391a0f98047 not found: ID does not exist" Oct 30 18:00:08.216691 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:08.216655 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-8pcnj"] Oct 30 18:00:08.220895 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:08.220871 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-8pcnj"] Oct 30 18:00:09.633135 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:09.633098 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="febf052c-1f46-4866-a1a3-88499118ae28" path="/var/lib/kubelet/pods/febf052c-1f46-4866-a1a3-88499118ae28/volumes" Oct 30 18:00:18.221450 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:18.221352 4455 generic.go:356] "Generic (PLEG): container finished" podID="d6fd4787-f2fc-4567-b223-8dcd356163cb" containerID="e20d7de49a7cb09745d84beaa1942e1251e2b150ee0cc267df23030baa34c8f1" exitCode=60 Oct 30 18:00:18.221450 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:18.221410 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-snxhj" event={"ID":"d6fd4787-f2fc-4567-b223-8dcd356163cb","Type":"ContainerDied","Data":"e20d7de49a7cb09745d84beaa1942e1251e2b150ee0cc267df23030baa34c8f1"} Oct 30 18:00:18.221897 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:18.221703 4455 scope.go:122] "RemoveContainer" containerID="e20d7de49a7cb09745d84beaa1942e1251e2b150ee0cc267df23030baa34c8f1" Oct 30 18:00:19.227929 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:19.227843 4455 generic.go:356] "Generic (PLEG): container finished" podID="d6fd4787-f2fc-4567-b223-8dcd356163cb" containerID="196d9f230b73c495452e4596acabf7648182b56815ee0a028b4d47f2b82de2d4" exitCode=60 Oct 30 18:00:19.227929 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:19.227919 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-snxhj" event={"ID":"d6fd4787-f2fc-4567-b223-8dcd356163cb","Type":"ContainerDied","Data":"196d9f230b73c495452e4596acabf7648182b56815ee0a028b4d47f2b82de2d4"} Oct 30 18:00:19.228427 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:19.227970 4455 scope.go:122] "RemoveContainer" containerID="e20d7de49a7cb09745d84beaa1942e1251e2b150ee0cc267df23030baa34c8f1" Oct 30 18:00:19.228427 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:19.228172 4455 scope.go:122] "RemoveContainer" containerID="196d9f230b73c495452e4596acabf7648182b56815ee0a028b4d47f2b82de2d4" Oct 30 18:00:19.228539 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:00:19.228519 4455 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889720-snxhj_odh-ai-gateway-infra(d6fd4787-f2fc-4567-b223-8dcd356163cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-snxhj" podUID="d6fd4787-f2fc-4567-b223-8dcd356163cb" Oct 30 18:00:20.234791 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:20.234764 4455 scope.go:122] "RemoveContainer" containerID="196d9f230b73c495452e4596acabf7648182b56815ee0a028b4d47f2b82de2d4" Oct 30 18:00:20.235171 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:00:20.234998 4455 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889720-snxhj_odh-ai-gateway-infra(d6fd4787-f2fc-4567-b223-8dcd356163cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-snxhj" podUID="d6fd4787-f2fc-4567-b223-8dcd356163cb" Oct 30 18:00:30.383925 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:00:30.383889 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:00:30.626801 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:30.626771 4455 scope.go:122] "RemoveContainer" containerID="196d9f230b73c495452e4596acabf7648182b56815ee0a028b4d47f2b82de2d4" Oct 30 18:00:31.297827 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:31.297738 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-snxhj" event={"ID":"d6fd4787-f2fc-4567-b223-8dcd356163cb","Type":"ContainerStarted","Data":"6717feb02e5fa804841dab5cdb910ce1c8f4dc99b059a8412d4781665d81f8fc"} Oct 30 18:00:31.338873 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:31.338807 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-snxhj" podStartSLOduration=14.923229379 podStartE2EDuration="31.338792474s" totalImagesPullingTime="16.415563095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:31.337198139 +0000 UTC m=+483.824835173" watchObservedRunningTime="2026-10-30 18:00:31.338792474 +0000 UTC m=+483.826429510" Oct 30 18:00:32.304642 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:32.304608 4455 generic.go:356] "Generic (PLEG): container finished" podID="d6fd4787-f2fc-4567-b223-8dcd356163cb" containerID="6717feb02e5fa804841dab5cdb910ce1c8f4dc99b059a8412d4781665d81f8fc" exitCode=60 Oct 30 18:00:32.305132 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:32.304658 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-snxhj" event={"ID":"d6fd4787-f2fc-4567-b223-8dcd356163cb","Type":"ContainerDied","Data":"6717feb02e5fa804841dab5cdb910ce1c8f4dc99b059a8412d4781665d81f8fc"} Oct 30 18:00:32.305132 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:32.304703 4455 scope.go:122] "RemoveContainer" containerID="196d9f230b73c495452e4596acabf7648182b56815ee0a028b4d47f2b82de2d4" Oct 30 18:00:32.305132 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:32.305012 4455 scope.go:122] "RemoveContainer" containerID="6717feb02e5fa804841dab5cdb910ce1c8f4dc99b059a8412d4781665d81f8fc" Oct 30 18:00:32.305265 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:00:32.305246 4455 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889720-snxhj_odh-ai-gateway-infra(d6fd4787-f2fc-4567-b223-8dcd356163cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-snxhj" podUID="d6fd4787-f2fc-4567-b223-8dcd356163cb" Oct 30 18:00:32.355911 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:32.355875 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889720-snxhj"] Oct 30 18:00:33.450611 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:33.450588 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-snxhj" Oct 30 18:00:33.586673 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:33.586587 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6fd4787-f2fc-4567-b223-8dcd356163cb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6fd4787-f2fc-4567-b223-8dcd356163cb-maas-api-service-ca\") pod \"d6fd4787-f2fc-4567-b223-8dcd356163cb\" (UID: \"d6fd4787-f2fc-4567-b223-8dcd356163cb\") " Oct 30 18:00:33.586673 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:33.586631 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6fd4787-f2fc-4567-b223-8dcd356163cb-kube-api-access-vtwg2\" (UniqueName: \"kubernetes.io/projected/d6fd4787-f2fc-4567-b223-8dcd356163cb-kube-api-access-vtwg2\") pod \"d6fd4787-f2fc-4567-b223-8dcd356163cb\" (UID: \"d6fd4787-f2fc-4567-b223-8dcd356163cb\") " Oct 30 18:00:33.586954 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:33.586928 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6fd4787-f2fc-4567-b223-8dcd356163cb-maas-api-service-ca" pod "d6fd4787-f2fc-4567-b223-8dcd356163cb" (UID: "d6fd4787-f2fc-4567-b223-8dcd356163cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:00:33.587041 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:33.587025 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6fd4787-f2fc-4567-b223-8dcd356163cb-maas-api-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:00:33.588747 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:33.588724 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6fd4787-f2fc-4567-b223-8dcd356163cb-kube-api-access-vtwg2" pod "d6fd4787-f2fc-4567-b223-8dcd356163cb" (UID: "d6fd4787-f2fc-4567-b223-8dcd356163cb"). InnerVolumeSpecName "kube-api-access-vtwg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:00:33.687515 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:33.687484 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtwg2\" (UniqueName: \"kubernetes.io/projected/d6fd4787-f2fc-4567-b223-8dcd356163cb-kube-api-access-vtwg2\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:00:34.326880 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:34.326847 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-snxhj" Oct 30 18:00:34.327069 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:34.326899 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889720-snxhj" event={"ID":"d6fd4787-f2fc-4567-b223-8dcd356163cb","Type":"ContainerDied","Data":"92c2d740d4876e4907a26b2ad1863864f20ae64f16845747bcfa9706a2f9b4a6"} Oct 30 18:00:34.327069 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:34.327031 4455 scope.go:122] "RemoveContainer" containerID="6717feb02e5fa804841dab5cdb910ce1c8f4dc99b059a8412d4781665d81f8fc" Oct 30 18:00:34.383469 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:34.383438 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889720-snxhj"] Oct 30 18:00:34.390210 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:34.390184 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889720-snxhj"] Oct 30 18:00:35.632366 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:35.632312 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6fd4787-f2fc-4567-b223-8dcd356163cb" path="/var/lib/kubelet/pods/d6fd4787-f2fc-4567-b223-8dcd356163cb/volumes" Oct 30 18:00:49.642415 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:49.642316 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm"] Oct 30 18:00:49.642904 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:49.642886 4455 cpu_manager.go:414] "containerMap: removing container" podUID="d6fd4787-f2fc-4567-b223-8dcd356163cb" containerName="cleanup" Oct 30 18:00:49.642947 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:49.642907 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6fd4787-f2fc-4567-b223-8dcd356163cb" containerName="cleanup" Oct 30 18:00:49.642947 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:49.642926 4455 cpu_manager.go:414] "containerMap: removing container" podUID="febf052c-1f46-4866-a1a3-88499118ae28" containerName="manager" Oct 30 18:00:49.642947 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:49.642931 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="febf052c-1f46-4866-a1a3-88499118ae28" containerName="manager" Oct 30 18:00:49.642947 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:49.642943 4455 cpu_manager.go:414] "containerMap: removing container" podUID="d6fd4787-f2fc-4567-b223-8dcd356163cb" containerName="cleanup" Oct 30 18:00:49.643067 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:49.642948 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6fd4787-f2fc-4567-b223-8dcd356163cb" containerName="cleanup" Oct 30 18:00:49.643067 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:49.643009 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6fd4787-f2fc-4567-b223-8dcd356163cb" containerName="cleanup" Oct 30 18:00:49.643067 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:49.643016 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="febf052c-1f46-4866-a1a3-88499118ae28" containerName="manager" Oct 30 18:00:49.643067 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:49.643028 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6fd4787-f2fc-4567-b223-8dcd356163cb" containerName="cleanup" Oct 30 18:00:49.646451 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:49.646431 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm" Oct 30 18:00:49.697557 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:49.697523 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm"] Oct 30 18:00:49.704640 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:49.704602 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 18:00:49.704798 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:49.704789 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 18:00:49.706435 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:49.706413 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-9lnst\"" Oct 30 18:00:49.706528 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:49.706492 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 18:00:49.739853 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:49.739821 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6e19981-c4a3-403e-9265-e33461b1d35a-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm\" (UID: \"e6e19981-c4a3-403e-9265-e33461b1d35a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm" Oct 30 18:00:49.740042 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:49.739877 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6e19981-c4a3-403e-9265-e33461b1d35a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm\" (UID: \"e6e19981-c4a3-403e-9265-e33461b1d35a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm" Oct 30 18:00:49.740109 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:49.740053 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6e19981-c4a3-403e-9265-e33461b1d35a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm\" (UID: \"e6e19981-c4a3-403e-9265-e33461b1d35a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm" Oct 30 18:00:49.740109 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:49.740099 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6e19981-c4a3-403e-9265-e33461b1d35a-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm\" (UID: \"e6e19981-c4a3-403e-9265-e33461b1d35a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm" Oct 30 18:00:49.740224 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:49.740207 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8ht2\" (UniqueName: \"kubernetes.io/projected/e6e19981-c4a3-403e-9265-e33461b1d35a-kube-api-access-x8ht2\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm\" (UID: \"e6e19981-c4a3-403e-9265-e33461b1d35a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm" Oct 30 18:00:49.740268 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:49.740239 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6e19981-c4a3-403e-9265-e33461b1d35a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm\" (UID: \"e6e19981-c4a3-403e-9265-e33461b1d35a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm" Oct 30 18:00:49.840903 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:49.840869 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8ht2\" (UniqueName: \"kubernetes.io/projected/e6e19981-c4a3-403e-9265-e33461b1d35a-kube-api-access-x8ht2\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm\" (UID: \"e6e19981-c4a3-403e-9265-e33461b1d35a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm" Oct 30 18:00:49.840903 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:49.840910 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6e19981-c4a3-403e-9265-e33461b1d35a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm\" (UID: \"e6e19981-c4a3-403e-9265-e33461b1d35a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm" Oct 30 18:00:49.841163 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:49.840965 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6e19981-c4a3-403e-9265-e33461b1d35a-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm\" (UID: \"e6e19981-c4a3-403e-9265-e33461b1d35a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm" Oct 30 18:00:49.841163 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:49.840990 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6e19981-c4a3-403e-9265-e33461b1d35a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm\" (UID: \"e6e19981-c4a3-403e-9265-e33461b1d35a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm" Oct 30 18:00:49.841163 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:49.841013 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6e19981-c4a3-403e-9265-e33461b1d35a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm\" (UID: \"e6e19981-c4a3-403e-9265-e33461b1d35a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm" Oct 30 18:00:49.841163 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:49.841035 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6e19981-c4a3-403e-9265-e33461b1d35a-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm\" (UID: \"e6e19981-c4a3-403e-9265-e33461b1d35a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm" Oct 30 18:00:49.841483 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:49.841462 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6e19981-c4a3-403e-9265-e33461b1d35a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm\" (UID: \"e6e19981-c4a3-403e-9265-e33461b1d35a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm" Oct 30 18:00:49.841542 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:49.841524 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6e19981-c4a3-403e-9265-e33461b1d35a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm\" (UID: \"e6e19981-c4a3-403e-9265-e33461b1d35a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm" Oct 30 18:00:49.841580 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:49.841543 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6e19981-c4a3-403e-9265-e33461b1d35a-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm\" (UID: \"e6e19981-c4a3-403e-9265-e33461b1d35a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm" Oct 30 18:00:49.843530 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:49.843508 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6e19981-c4a3-403e-9265-e33461b1d35a-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm\" (UID: \"e6e19981-c4a3-403e-9265-e33461b1d35a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm" Oct 30 18:00:49.843793 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:49.843775 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6e19981-c4a3-403e-9265-e33461b1d35a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm\" (UID: \"e6e19981-c4a3-403e-9265-e33461b1d35a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm" Oct 30 18:00:49.863932 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:49.863906 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8ht2\" (UniqueName: \"kubernetes.io/projected/e6e19981-c4a3-403e-9265-e33461b1d35a-kube-api-access-x8ht2\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm\" (UID: \"e6e19981-c4a3-403e-9265-e33461b1d35a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm" Oct 30 18:00:49.958940 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:49.958849 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm" Oct 30 18:00:50.259338 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:50.259288 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm"] Oct 30 18:00:50.260991 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:00:50.260956 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6e19981_c4a3_403e_9265_e33461b1d35a.slice/crio-edc7fc236428fb0778763967f9d7186c66efc07975c72f18da4658796b80a9ff WatchSource:0}: Error finding container edc7fc236428fb0778763967f9d7186c66efc07975c72f18da4658796b80a9ff: Status 404 returned error can't find the container with id edc7fc236428fb0778763967f9d7186c66efc07975c72f18da4658796b80a9ff Oct 30 18:00:50.414479 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:50.414446 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm" event={"ID":"e6e19981-c4a3-403e-9265-e33461b1d35a","Type":"ContainerStarted","Data":"edc7fc236428fb0778763967f9d7186c66efc07975c72f18da4658796b80a9ff"} Oct 30 18:00:52.449748 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:52.449713 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm" event={"ID":"e6e19981-c4a3-403e-9265-e33461b1d35a","Type":"ContainerStarted","Data":"4062a5d7bce3aa218fb78d059ade6468ad524b4f2d2cecbb3dc752a6eab1c3e1"} Oct 30 18:00:52.492480 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:52.492424 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm" podStartSLOduration=1.948308146 podStartE2EDuration="3.492409741s" totalImagesPullingTime="1.544101595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:52.491884189 +0000 UTC m=+504.979521223" watchObservedRunningTime="2026-10-30 18:00:52.492409741 +0000 UTC m=+504.980046775" Oct 30 18:00:52.959445 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:52.959407 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm" Oct 30 18:00:57.125457 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:57.125420 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-nj9sw"] Oct 30 18:00:57.126204 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:57.126180 4455 cpu_manager.go:414] "containerMap: removing container" podUID="d6fd4787-f2fc-4567-b223-8dcd356163cb" containerName="cleanup" Oct 30 18:00:57.126374 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:57.126206 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6fd4787-f2fc-4567-b223-8dcd356163cb" containerName="cleanup" Oct 30 18:00:57.126448 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:57.126389 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6fd4787-f2fc-4567-b223-8dcd356163cb" containerName="cleanup" Oct 30 18:00:57.135368 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:57.129255 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nj9sw" Oct 30 18:00:57.144052 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:57.144024 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 18:00:57.163291 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:57.163259 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-nj9sw"] Oct 30 18:00:57.219220 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:57.219174 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efdc3c98-3ce3-4b85-90c1-27984b259887-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nj9sw\" (UID: \"efdc3c98-3ce3-4b85-90c1-27984b259887\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nj9sw" Oct 30 18:00:57.221192 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:57.221164 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efdc3c98-3ce3-4b85-90c1-27984b259887-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nj9sw\" (UID: \"efdc3c98-3ce3-4b85-90c1-27984b259887\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nj9sw" Oct 30 18:00:57.221508 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:57.221490 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efdc3c98-3ce3-4b85-90c1-27984b259887-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nj9sw\" (UID: \"efdc3c98-3ce3-4b85-90c1-27984b259887\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nj9sw" Oct 30 18:00:57.221710 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:57.221693 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd7l2\" (UniqueName: \"kubernetes.io/projected/efdc3c98-3ce3-4b85-90c1-27984b259887-kube-api-access-pd7l2\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nj9sw\" (UID: \"efdc3c98-3ce3-4b85-90c1-27984b259887\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nj9sw" Oct 30 18:00:57.221893 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:57.221880 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efdc3c98-3ce3-4b85-90c1-27984b259887-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nj9sw\" (UID: \"efdc3c98-3ce3-4b85-90c1-27984b259887\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nj9sw" Oct 30 18:00:57.222036 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:57.222024 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efdc3c98-3ce3-4b85-90c1-27984b259887-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nj9sw\" (UID: \"efdc3c98-3ce3-4b85-90c1-27984b259887\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nj9sw" Oct 30 18:00:57.333883 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:57.333832 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efdc3c98-3ce3-4b85-90c1-27984b259887-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nj9sw\" (UID: \"efdc3c98-3ce3-4b85-90c1-27984b259887\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nj9sw" Oct 30 18:00:57.334894 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:57.334856 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efdc3c98-3ce3-4b85-90c1-27984b259887-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nj9sw\" (UID: \"efdc3c98-3ce3-4b85-90c1-27984b259887\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nj9sw" Oct 30 18:00:57.335752 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:57.335484 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efdc3c98-3ce3-4b85-90c1-27984b259887-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nj9sw\" (UID: \"efdc3c98-3ce3-4b85-90c1-27984b259887\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nj9sw" Oct 30 18:00:57.335752 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:57.335587 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd7l2\" (UniqueName: \"kubernetes.io/projected/efdc3c98-3ce3-4b85-90c1-27984b259887-kube-api-access-pd7l2\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nj9sw\" (UID: \"efdc3c98-3ce3-4b85-90c1-27984b259887\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nj9sw" Oct 30 18:00:57.335752 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:57.335656 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efdc3c98-3ce3-4b85-90c1-27984b259887-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nj9sw\" (UID: \"efdc3c98-3ce3-4b85-90c1-27984b259887\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nj9sw" Oct 30 18:00:57.335752 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:57.335702 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efdc3c98-3ce3-4b85-90c1-27984b259887-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nj9sw\" (UID: \"efdc3c98-3ce3-4b85-90c1-27984b259887\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nj9sw" Oct 30 18:00:57.336177 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:57.336153 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efdc3c98-3ce3-4b85-90c1-27984b259887-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nj9sw\" (UID: \"efdc3c98-3ce3-4b85-90c1-27984b259887\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nj9sw" Oct 30 18:00:57.336404 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:57.334747 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efdc3c98-3ce3-4b85-90c1-27984b259887-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nj9sw\" (UID: \"efdc3c98-3ce3-4b85-90c1-27984b259887\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nj9sw" Oct 30 18:00:57.336495 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:57.336471 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efdc3c98-3ce3-4b85-90c1-27984b259887-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nj9sw\" (UID: \"efdc3c98-3ce3-4b85-90c1-27984b259887\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nj9sw" Oct 30 18:00:57.340389 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:57.340363 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efdc3c98-3ce3-4b85-90c1-27984b259887-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nj9sw\" (UID: \"efdc3c98-3ce3-4b85-90c1-27984b259887\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nj9sw" Oct 30 18:00:57.340745 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:57.340724 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efdc3c98-3ce3-4b85-90c1-27984b259887-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nj9sw\" (UID: \"efdc3c98-3ce3-4b85-90c1-27984b259887\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nj9sw" Oct 30 18:00:57.359068 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:57.359036 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd7l2\" (UniqueName: \"kubernetes.io/projected/efdc3c98-3ce3-4b85-90c1-27984b259887-kube-api-access-pd7l2\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nj9sw\" (UID: \"efdc3c98-3ce3-4b85-90c1-27984b259887\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nj9sw" Oct 30 18:00:57.451770 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:57.451678 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nj9sw" Oct 30 18:00:57.808926 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:57.808892 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-nj9sw"] Oct 30 18:00:57.811291 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:00:57.811255 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefdc3c98_3ce3_4b85_90c1_27984b259887.slice/crio-9b778d6fface5b1722e1b0b65182d2789e8a0c59cb99e31102ea54205e6ca8e4 WatchSource:0}: Error finding container 9b778d6fface5b1722e1b0b65182d2789e8a0c59cb99e31102ea54205e6ca8e4: Status 404 returned error can't find the container with id 9b778d6fface5b1722e1b0b65182d2789e8a0c59cb99e31102ea54205e6ca8e4 Oct 30 18:00:58.550956 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:58.550919 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nj9sw" event={"ID":"efdc3c98-3ce3-4b85-90c1-27984b259887","Type":"ContainerStarted","Data":"6510970d35fcb9dcbd3e570d0fe6dac8e60d1acf2608d92f0b160575d4f80c08"} Oct 30 18:00:58.551498 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:58.550970 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nj9sw" event={"ID":"efdc3c98-3ce3-4b85-90c1-27984b259887","Type":"ContainerStarted","Data":"9b778d6fface5b1722e1b0b65182d2789e8a0c59cb99e31102ea54205e6ca8e4"} Oct 30 18:00:58.596776 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:58.596716 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nj9sw" podStartSLOduration=1.411600011 podStartE2EDuration="1.596697708s" totalImagesPullingTime="185.097697ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:58.594996451 +0000 UTC m=+511.082633497" watchObservedRunningTime="2026-10-30 18:00:58.596697708 +0000 UTC m=+511.084334744" Oct 30 18:00:59.452348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:59.451815 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nj9sw" Oct 30 18:00:59.959305 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:59.959268 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm" Oct 30 18:00:59.982661 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:00:59.981053 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm" Oct 30 18:01:00.396472 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:01:00.396430 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:01:00.565761 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:00.565727 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc"] Oct 30 18:01:00.570532 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:00.570509 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc" Oct 30 18:01:00.588172 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:00.588077 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm" Oct 30 18:01:00.588861 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:00.588624 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 30 18:01:00.604021 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:00.602487 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc"] Oct 30 18:01:00.684687 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:00.684593 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7222c61-e786-4955-bd32-5f90a63ea48b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc\" (UID: \"b7222c61-e786-4955-bd32-5f90a63ea48b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc" Oct 30 18:01:00.684864 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:00.684696 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7222c61-e786-4955-bd32-5f90a63ea48b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc\" (UID: \"b7222c61-e786-4955-bd32-5f90a63ea48b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc" Oct 30 18:01:00.684864 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:00.684818 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bwgw\" (UniqueName: \"kubernetes.io/projected/b7222c61-e786-4955-bd32-5f90a63ea48b-kube-api-access-6bwgw\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc\" (UID: \"b7222c61-e786-4955-bd32-5f90a63ea48b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc" Oct 30 18:01:00.684864 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:00.684856 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7222c61-e786-4955-bd32-5f90a63ea48b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc\" (UID: \"b7222c61-e786-4955-bd32-5f90a63ea48b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc" Oct 30 18:01:00.685077 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:00.685053 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7222c61-e786-4955-bd32-5f90a63ea48b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc\" (UID: \"b7222c61-e786-4955-bd32-5f90a63ea48b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc" Oct 30 18:01:00.685142 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:00.685116 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7222c61-e786-4955-bd32-5f90a63ea48b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc\" (UID: \"b7222c61-e786-4955-bd32-5f90a63ea48b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc" Oct 30 18:01:00.786072 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:00.786040 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7222c61-e786-4955-bd32-5f90a63ea48b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc\" (UID: \"b7222c61-e786-4955-bd32-5f90a63ea48b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc" Oct 30 18:01:00.786072 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:00.786074 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7222c61-e786-4955-bd32-5f90a63ea48b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc\" (UID: \"b7222c61-e786-4955-bd32-5f90a63ea48b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc" Oct 30 18:01:00.786304 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:00.786126 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7222c61-e786-4955-bd32-5f90a63ea48b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc\" (UID: \"b7222c61-e786-4955-bd32-5f90a63ea48b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc" Oct 30 18:01:00.786304 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:00.786151 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7222c61-e786-4955-bd32-5f90a63ea48b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc\" (UID: \"b7222c61-e786-4955-bd32-5f90a63ea48b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc" Oct 30 18:01:00.786304 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:00.786181 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bwgw\" (UniqueName: \"kubernetes.io/projected/b7222c61-e786-4955-bd32-5f90a63ea48b-kube-api-access-6bwgw\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc\" (UID: \"b7222c61-e786-4955-bd32-5f90a63ea48b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc" Oct 30 18:01:00.786304 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:00.786207 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7222c61-e786-4955-bd32-5f90a63ea48b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc\" (UID: \"b7222c61-e786-4955-bd32-5f90a63ea48b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc" Oct 30 18:01:00.786545 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:00.786512 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7222c61-e786-4955-bd32-5f90a63ea48b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc\" (UID: \"b7222c61-e786-4955-bd32-5f90a63ea48b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc" Oct 30 18:01:00.786612 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:00.786548 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7222c61-e786-4955-bd32-5f90a63ea48b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc\" (UID: \"b7222c61-e786-4955-bd32-5f90a63ea48b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc" Oct 30 18:01:00.786669 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:00.786620 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7222c61-e786-4955-bd32-5f90a63ea48b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc\" (UID: \"b7222c61-e786-4955-bd32-5f90a63ea48b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc" Oct 30 18:01:00.788608 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:00.788583 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7222c61-e786-4955-bd32-5f90a63ea48b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc\" (UID: \"b7222c61-e786-4955-bd32-5f90a63ea48b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc" Oct 30 18:01:00.788898 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:00.788882 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7222c61-e786-4955-bd32-5f90a63ea48b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc\" (UID: \"b7222c61-e786-4955-bd32-5f90a63ea48b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc" Oct 30 18:01:00.809054 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:00.809021 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bwgw\" (UniqueName: \"kubernetes.io/projected/b7222c61-e786-4955-bd32-5f90a63ea48b-kube-api-access-6bwgw\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc\" (UID: \"b7222c61-e786-4955-bd32-5f90a63ea48b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc" Oct 30 18:01:00.891007 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:00.890976 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc" Oct 30 18:01:01.191074 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:01.190959 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc"] Oct 30 18:01:01.193563 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:01:01.193535 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7222c61_e786_4955_bd32_5f90a63ea48b.slice/crio-cac458ec2c59e2109aa542d5acbbdce5acd65e75fef36e02e0bf9c58d5428aa6 WatchSource:0}: Error finding container cac458ec2c59e2109aa542d5acbbdce5acd65e75fef36e02e0bf9c58d5428aa6: Status 404 returned error can't find the container with id cac458ec2c59e2109aa542d5acbbdce5acd65e75fef36e02e0bf9c58d5428aa6 Oct 30 18:01:01.576235 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:01.576193 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc" event={"ID":"b7222c61-e786-4955-bd32-5f90a63ea48b","Type":"ContainerStarted","Data":"cd3ff5543e2790509c574c6ca533182be40dfae6b6da0a3d77ccf55c74156c81"} Oct 30 18:01:01.576549 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:01.576256 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc" event={"ID":"b7222c61-e786-4955-bd32-5f90a63ea48b","Type":"ContainerStarted","Data":"cac458ec2c59e2109aa542d5acbbdce5acd65e75fef36e02e0bf9c58d5428aa6"} Oct 30 18:01:01.618584 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:01.618520 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc" podStartSLOduration=1.396019893 podStartE2EDuration="1.618504694s" totalImagesPullingTime="222.484801ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:01.61672551 +0000 UTC m=+514.104362557" watchObservedRunningTime="2026-10-30 18:01:01.618504694 +0000 UTC m=+514.106141752" Oct 30 18:01:01.891953 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:01.891915 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc" Oct 30 18:01:03.930356 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:03.929557 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9"] Oct 30 18:01:03.942123 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:03.942094 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9" Oct 30 18:01:03.956538 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:03.956516 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 18:01:03.984935 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:03.978221 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9"] Oct 30 18:01:04.041639 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:04.041599 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/484dd1d5-ee6a-4423-87c4-efbb4ba01767-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9\" (UID: \"484dd1d5-ee6a-4423-87c4-efbb4ba01767\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9" Oct 30 18:01:04.041802 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:04.041647 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/484dd1d5-ee6a-4423-87c4-efbb4ba01767-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9\" (UID: \"484dd1d5-ee6a-4423-87c4-efbb4ba01767\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9" Oct 30 18:01:04.041848 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:04.041826 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/484dd1d5-ee6a-4423-87c4-efbb4ba01767-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9\" (UID: \"484dd1d5-ee6a-4423-87c4-efbb4ba01767\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9" Oct 30 18:01:04.041894 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:04.041877 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/484dd1d5-ee6a-4423-87c4-efbb4ba01767-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9\" (UID: \"484dd1d5-ee6a-4423-87c4-efbb4ba01767\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9" Oct 30 18:01:04.041981 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:04.041963 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtnwm\" (UniqueName: \"kubernetes.io/projected/484dd1d5-ee6a-4423-87c4-efbb4ba01767-kube-api-access-gtnwm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9\" (UID: \"484dd1d5-ee6a-4423-87c4-efbb4ba01767\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9" Oct 30 18:01:04.042067 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:04.042053 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/484dd1d5-ee6a-4423-87c4-efbb4ba01767-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9\" (UID: \"484dd1d5-ee6a-4423-87c4-efbb4ba01767\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9" Oct 30 18:01:04.142945 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:04.142916 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/484dd1d5-ee6a-4423-87c4-efbb4ba01767-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9\" (UID: \"484dd1d5-ee6a-4423-87c4-efbb4ba01767\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9" Oct 30 18:01:04.143166 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:04.142970 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/484dd1d5-ee6a-4423-87c4-efbb4ba01767-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9\" (UID: \"484dd1d5-ee6a-4423-87c4-efbb4ba01767\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9" Oct 30 18:01:04.143166 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:04.143002 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtnwm\" (UniqueName: \"kubernetes.io/projected/484dd1d5-ee6a-4423-87c4-efbb4ba01767-kube-api-access-gtnwm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9\" (UID: \"484dd1d5-ee6a-4423-87c4-efbb4ba01767\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9" Oct 30 18:01:04.143166 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:04.143046 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/484dd1d5-ee6a-4423-87c4-efbb4ba01767-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9\" (UID: \"484dd1d5-ee6a-4423-87c4-efbb4ba01767\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9" Oct 30 18:01:04.143374 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:04.143169 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/484dd1d5-ee6a-4423-87c4-efbb4ba01767-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9\" (UID: \"484dd1d5-ee6a-4423-87c4-efbb4ba01767\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9" Oct 30 18:01:04.143374 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:04.143219 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/484dd1d5-ee6a-4423-87c4-efbb4ba01767-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9\" (UID: \"484dd1d5-ee6a-4423-87c4-efbb4ba01767\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9" Oct 30 18:01:04.143493 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:04.143393 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/484dd1d5-ee6a-4423-87c4-efbb4ba01767-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9\" (UID: \"484dd1d5-ee6a-4423-87c4-efbb4ba01767\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9" Oct 30 18:01:04.143493 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:04.143483 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/484dd1d5-ee6a-4423-87c4-efbb4ba01767-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9\" (UID: \"484dd1d5-ee6a-4423-87c4-efbb4ba01767\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9" Oct 30 18:01:04.143704 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:04.143683 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/484dd1d5-ee6a-4423-87c4-efbb4ba01767-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9\" (UID: \"484dd1d5-ee6a-4423-87c4-efbb4ba01767\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9" Oct 30 18:01:04.145562 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:04.145547 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/484dd1d5-ee6a-4423-87c4-efbb4ba01767-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9\" (UID: \"484dd1d5-ee6a-4423-87c4-efbb4ba01767\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9" Oct 30 18:01:04.145724 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:04.145708 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/484dd1d5-ee6a-4423-87c4-efbb4ba01767-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9\" (UID: \"484dd1d5-ee6a-4423-87c4-efbb4ba01767\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9" Oct 30 18:01:04.184811 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:04.184714 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtnwm\" (UniqueName: \"kubernetes.io/projected/484dd1d5-ee6a-4423-87c4-efbb4ba01767-kube-api-access-gtnwm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9\" (UID: \"484dd1d5-ee6a-4423-87c4-efbb4ba01767\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9" Oct 30 18:01:04.258043 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:04.258004 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9" Oct 30 18:01:04.573606 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:04.573578 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9"] Oct 30 18:01:04.576282 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:01:04.576254 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod484dd1d5_ee6a_4423_87c4_efbb4ba01767.slice/crio-cde633ecc31f8d1e24a2b487335f06c65b7c603b74f26dff4538820d7a17b8c8 WatchSource:0}: Error finding container cde633ecc31f8d1e24a2b487335f06c65b7c603b74f26dff4538820d7a17b8c8: Status 404 returned error can't find the container with id cde633ecc31f8d1e24a2b487335f06c65b7c603b74f26dff4538820d7a17b8c8 Oct 30 18:01:04.607442 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:04.607408 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9" event={"ID":"484dd1d5-ee6a-4423-87c4-efbb4ba01767","Type":"ContainerStarted","Data":"cde633ecc31f8d1e24a2b487335f06c65b7c603b74f26dff4538820d7a17b8c8"} Oct 30 18:01:05.615386 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:05.615352 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9" event={"ID":"484dd1d5-ee6a-4423-87c4-efbb4ba01767","Type":"ContainerStarted","Data":"a3768e07dad8cf1b6a20c30128958cf3aca0791f7b69e0582cb436bd46a12cfc"} Oct 30 18:01:05.663067 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:05.663012 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9" podStartSLOduration=2.439666436 podStartE2EDuration="2.662995631s" totalImagesPullingTime="223.329195ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:05.660032559 +0000 UTC m=+518.147669594" watchObservedRunningTime="2026-10-30 18:01:05.662995631 +0000 UTC m=+518.150632665" Oct 30 18:01:06.258898 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:06.258855 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9" Oct 30 18:01:07.452353 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:07.452293 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nj9sw" Oct 30 18:01:07.464991 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:07.464966 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nj9sw" Oct 30 18:01:07.651399 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:07.651363 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nj9sw" Oct 30 18:01:08.391905 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:08.391866 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2"] Oct 30 18:01:08.396762 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:08.396736 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" Oct 30 18:01:08.445352 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:08.444962 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2"] Oct 30 18:01:08.541993 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:08.541955 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82d99e75-d468-4b37-b42e-391602a39c85-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-9kmd2\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" Oct 30 18:01:08.542443 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:08.542003 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82d99e75-d468-4b37-b42e-391602a39c85-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-9kmd2\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" Oct 30 18:01:08.542443 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:08.542063 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d99e75-d468-4b37-b42e-391602a39c85-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9kmd2\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" Oct 30 18:01:08.542443 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:08.542107 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82d99e75-d468-4b37-b42e-391602a39c85-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-9kmd2\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" Oct 30 18:01:08.542443 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:08.542254 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d99e75-d468-4b37-b42e-391602a39c85-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9kmd2\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" Oct 30 18:01:08.542443 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:08.542281 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82d99e75-d468-4b37-b42e-391602a39c85-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-9kmd2\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" Oct 30 18:01:08.542443 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:08.542303 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82d99e75-d468-4b37-b42e-391602a39c85-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-9kmd2\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" Oct 30 18:01:08.542443 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:08.542425 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-286qq\" (UniqueName: \"kubernetes.io/projected/82d99e75-d468-4b37-b42e-391602a39c85-kube-api-access-286qq\") pod \"maas-default-gateway-openshift-default-67bf986f99-9kmd2\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" Oct 30 18:01:08.542672 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:08.542479 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82d99e75-d468-4b37-b42e-391602a39c85-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-9kmd2\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" Oct 30 18:01:08.643600 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:08.643520 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-286qq\" (UniqueName: \"kubernetes.io/projected/82d99e75-d468-4b37-b42e-391602a39c85-kube-api-access-286qq\") pod \"maas-default-gateway-openshift-default-67bf986f99-9kmd2\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" Oct 30 18:01:08.643600 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:08.643566 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82d99e75-d468-4b37-b42e-391602a39c85-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-9kmd2\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" Oct 30 18:01:08.643816 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:08.643609 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82d99e75-d468-4b37-b42e-391602a39c85-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-9kmd2\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" Oct 30 18:01:08.643816 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:08.643651 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82d99e75-d468-4b37-b42e-391602a39c85-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-9kmd2\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" Oct 30 18:01:08.643816 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:08.643680 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d99e75-d468-4b37-b42e-391602a39c85-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9kmd2\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" Oct 30 18:01:08.643816 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:08.643702 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82d99e75-d468-4b37-b42e-391602a39c85-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-9kmd2\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" Oct 30 18:01:08.643816 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:08.643754 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d99e75-d468-4b37-b42e-391602a39c85-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9kmd2\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" Oct 30 18:01:08.643816 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:08.643779 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82d99e75-d468-4b37-b42e-391602a39c85-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-9kmd2\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" Oct 30 18:01:08.643816 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:08.643809 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82d99e75-d468-4b37-b42e-391602a39c85-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-9kmd2\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" Oct 30 18:01:08.644078 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:08.644004 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82d99e75-d468-4b37-b42e-391602a39c85-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-9kmd2\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" Oct 30 18:01:08.644138 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:08.644101 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82d99e75-d468-4b37-b42e-391602a39c85-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-9kmd2\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" Oct 30 18:01:08.644608 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:08.644570 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d99e75-d468-4b37-b42e-391602a39c85-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9kmd2\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" Oct 30 18:01:08.644724 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:08.644673 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82d99e75-d468-4b37-b42e-391602a39c85-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-9kmd2\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" Oct 30 18:01:08.644784 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:08.644759 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d99e75-d468-4b37-b42e-391602a39c85-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9kmd2\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" Oct 30 18:01:08.646594 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:08.646574 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82d99e75-d468-4b37-b42e-391602a39c85-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-9kmd2\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" Oct 30 18:01:08.647099 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:08.647080 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82d99e75-d468-4b37-b42e-391602a39c85-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-9kmd2\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" Oct 30 18:01:08.668091 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:08.668068 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-286qq\" (UniqueName: \"kubernetes.io/projected/82d99e75-d468-4b37-b42e-391602a39c85-kube-api-access-286qq\") pod \"maas-default-gateway-openshift-default-67bf986f99-9kmd2\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" Oct 30 18:01:08.680406 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:08.671550 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82d99e75-d468-4b37-b42e-391602a39c85-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-9kmd2\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" Oct 30 18:01:08.756049 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:08.756016 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" Oct 30 18:01:09.097908 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:09.097879 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2"] Oct 30 18:01:09.099496 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:01:09.099467 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d99e75_d468_4b37_b42e_391602a39c85.slice/crio-75457c30c16efebbad64a2083831ba1c93cf82853f7935b495c957ce5a31c1ab WatchSource:0}: Error finding container 75457c30c16efebbad64a2083831ba1c93cf82853f7935b495c957ce5a31c1ab: Status 404 returned error can't find the container with id 75457c30c16efebbad64a2083831ba1c93cf82853f7935b495c957ce5a31c1ab Oct 30 18:01:09.102280 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:09.102246 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:01:09.102404 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:09.102314 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:01:09.102404 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:09.102369 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:01:09.652519 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:09.652387 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" event={"ID":"82d99e75-d468-4b37-b42e-391602a39c85","Type":"ContainerStarted","Data":"af862c6c1d1eadc202152dada9a4c2b46f7dd30fde559a5586e11fbcd6592a52"} Oct 30 18:01:09.652519 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:09.652467 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" event={"ID":"82d99e75-d468-4b37-b42e-391602a39c85","Type":"ContainerStarted","Data":"75457c30c16efebbad64a2083831ba1c93cf82853f7935b495c957ce5a31c1ab"} Oct 30 18:01:09.734349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:09.729989 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" podStartSLOduration=1.7299661400000002 podStartE2EDuration="1.72996614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:09.716234969 +0000 UTC m=+522.203872004" watchObservedRunningTime="2026-10-30 18:01:09.72996614 +0000 UTC m=+522.217603203" Oct 30 18:01:09.756281 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:09.756234 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" Oct 30 18:01:10.758234 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:10.758199 4455 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9kmd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 18:01:10.758701 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:10.758259 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" podUID="82d99e75-d468-4b37-b42e-391602a39c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 18:01:10.892542 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:10.892502 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc" Oct 30 18:01:10.910379 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:10.910349 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc" Oct 30 18:01:11.697613 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:11.697580 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc" Oct 30 18:01:11.757409 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:11.757374 4455 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9kmd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 18:01:11.757574 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:11.757448 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" podUID="82d99e75-d468-4b37-b42e-391602a39c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 18:01:12.756737 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:12.756706 4455 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9kmd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 18:01:12.757154 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:12.756768 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" podUID="82d99e75-d468-4b37-b42e-391602a39c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 18:01:13.756505 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:13.756472 4455 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9kmd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 18:01:13.756677 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:13.756532 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" podUID="82d99e75-d468-4b37-b42e-391602a39c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 18:01:14.258909 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:14.258876 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9" Oct 30 18:01:14.288349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:14.287702 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9" Oct 30 18:01:14.714753 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:14.714671 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9" Oct 30 18:01:14.757535 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:14.757500 4455 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9kmd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 18:01:14.757820 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:14.757785 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" podUID="82d99e75-d468-4b37-b42e-391602a39c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 18:01:15.757191 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:15.757156 4455 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9kmd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 18:01:15.757615 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:15.757209 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" podUID="82d99e75-d468-4b37-b42e-391602a39c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 18:01:16.756832 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:16.756799 4455 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9kmd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 18:01:16.757021 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:16.756852 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" podUID="82d99e75-d468-4b37-b42e-391602a39c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 18:01:17.762618 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:17.762585 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" Oct 30 18:01:17.763121 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:17.763004 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" Oct 30 18:01:17.763881 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:17.763862 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" Oct 30 18:01:30.384678 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:01:30.384641 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:01:31.678235 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:31.678201 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d56799cb-n8rw5"] Oct 30 18:01:31.682821 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:31.682797 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d56799cb-n8rw5" Oct 30 18:01:31.702873 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:31.702835 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d56799cb-n8rw5"] Oct 30 18:01:31.805101 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:31.805069 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9eea1448-1cc7-4ff1-b584-2b7f3e1b730d-tls-cert\") pod \"authorino-75d56799cb-n8rw5\" (UID: \"9eea1448-1cc7-4ff1-b584-2b7f3e1b730d\") " pod="kuadrant-system/authorino-75d56799cb-n8rw5" Oct 30 18:01:31.805277 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:31.805143 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm2fk\" (UniqueName: \"kubernetes.io/projected/9eea1448-1cc7-4ff1-b584-2b7f3e1b730d-kube-api-access-jm2fk\") pod \"authorino-75d56799cb-n8rw5\" (UID: \"9eea1448-1cc7-4ff1-b584-2b7f3e1b730d\") " pod="kuadrant-system/authorino-75d56799cb-n8rw5" Oct 30 18:01:31.906428 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:31.906389 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm2fk\" (UniqueName: \"kubernetes.io/projected/9eea1448-1cc7-4ff1-b584-2b7f3e1b730d-kube-api-access-jm2fk\") pod \"authorino-75d56799cb-n8rw5\" (UID: \"9eea1448-1cc7-4ff1-b584-2b7f3e1b730d\") " pod="kuadrant-system/authorino-75d56799cb-n8rw5" Oct 30 18:01:31.906608 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:31.906506 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9eea1448-1cc7-4ff1-b584-2b7f3e1b730d-tls-cert\") pod \"authorino-75d56799cb-n8rw5\" (UID: \"9eea1448-1cc7-4ff1-b584-2b7f3e1b730d\") " pod="kuadrant-system/authorino-75d56799cb-n8rw5" Oct 30 18:01:31.909139 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:31.909114 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9eea1448-1cc7-4ff1-b584-2b7f3e1b730d-tls-cert\") pod \"authorino-75d56799cb-n8rw5\" (UID: \"9eea1448-1cc7-4ff1-b584-2b7f3e1b730d\") " pod="kuadrant-system/authorino-75d56799cb-n8rw5" Oct 30 18:01:31.929377 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:31.929294 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm2fk\" (UniqueName: \"kubernetes.io/projected/9eea1448-1cc7-4ff1-b584-2b7f3e1b730d-kube-api-access-jm2fk\") pod \"authorino-75d56799cb-n8rw5\" (UID: \"9eea1448-1cc7-4ff1-b584-2b7f3e1b730d\") " pod="kuadrant-system/authorino-75d56799cb-n8rw5" Oct 30 18:01:31.996337 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:31.996284 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d56799cb-n8rw5" Oct 30 18:01:32.291289 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:32.290868 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d56799cb-n8rw5"] Oct 30 18:01:32.293453 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:01:32.293421 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9eea1448_1cc7_4ff1_b584_2b7f3e1b730d.slice/crio-e610c8470a7e8b0ac040345f0954eb28ef4f3aa17a912f63931fa5c35fa7fc40 WatchSource:0}: Error finding container e610c8470a7e8b0ac040345f0954eb28ef4f3aa17a912f63931fa5c35fa7fc40: Status 404 returned error can't find the container with id e610c8470a7e8b0ac040345f0954eb28ef4f3aa17a912f63931fa5c35fa7fc40 Oct 30 18:01:32.812615 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:32.812581 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d56799cb-n8rw5" event={"ID":"9eea1448-1cc7-4ff1-b584-2b7f3e1b730d","Type":"ContainerStarted","Data":"e610c8470a7e8b0ac040345f0954eb28ef4f3aa17a912f63931fa5c35fa7fc40"} Oct 30 18:01:33.834058 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:33.834022 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d56799cb-n8rw5" event={"ID":"9eea1448-1cc7-4ff1-b584-2b7f3e1b730d","Type":"ContainerStarted","Data":"ef71374e83aeecac97a66d3418215699633e59eba984b2f1c4d236735439b6c8"} Oct 30 18:01:33.886512 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:33.886438 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-75d56799cb-n8rw5" podStartSLOduration=2.126465807 podStartE2EDuration="2.886417583s" totalImagesPullingTime="759.951776ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:33.881203291 +0000 UTC m=+546.368840328" watchObservedRunningTime="2026-10-30 18:01:33.886417583 +0000 UTC m=+546.374054624" Oct 30 18:01:33.913540 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:33.913505 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-55b8fd5845-fdl6z"] Oct 30 18:01:33.914055 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:33.914020 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-55b8fd5845-fdl6z" podUID="6e924469-24ea-4da5-9658-2c35efdea8dd" containerName="authorino" containerID="cri-o://6c501214c9bbb3d54b536a2ec92649f9979f929ecd39a0c52f794d63da79b2d3" gracePeriod=30 Oct 30 18:01:34.211930 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:34.211900 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55b8fd5845-fdl6z" Oct 30 18:01:34.340610 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:34.339703 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e924469-24ea-4da5-9658-2c35efdea8dd-tls-cert\" (UniqueName: \"kubernetes.io/secret/6e924469-24ea-4da5-9658-2c35efdea8dd-tls-cert\") pod \"6e924469-24ea-4da5-9658-2c35efdea8dd\" (UID: \"6e924469-24ea-4da5-9658-2c35efdea8dd\") " Oct 30 18:01:34.340610 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:34.339769 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e924469-24ea-4da5-9658-2c35efdea8dd-kube-api-access-6xzkp\" (UniqueName: \"kubernetes.io/projected/6e924469-24ea-4da5-9658-2c35efdea8dd-kube-api-access-6xzkp\") pod \"6e924469-24ea-4da5-9658-2c35efdea8dd\" (UID: \"6e924469-24ea-4da5-9658-2c35efdea8dd\") " Oct 30 18:01:34.344164 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:34.344134 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e924469-24ea-4da5-9658-2c35efdea8dd-kube-api-access-6xzkp" pod "6e924469-24ea-4da5-9658-2c35efdea8dd" (UID: "6e924469-24ea-4da5-9658-2c35efdea8dd"). InnerVolumeSpecName "kube-api-access-6xzkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:01:34.358051 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:34.358014 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e924469-24ea-4da5-9658-2c35efdea8dd-tls-cert" pod "6e924469-24ea-4da5-9658-2c35efdea8dd" (UID: "6e924469-24ea-4da5-9658-2c35efdea8dd"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:01:34.441192 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:34.441157 4455 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6e924469-24ea-4da5-9658-2c35efdea8dd-tls-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:01:34.441192 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:34.441186 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xzkp\" (UniqueName: \"kubernetes.io/projected/6e924469-24ea-4da5-9658-2c35efdea8dd-kube-api-access-6xzkp\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:01:34.842638 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:34.842606 4455 generic.go:356] "Generic (PLEG): container finished" podID="6e924469-24ea-4da5-9658-2c35efdea8dd" containerID="6c501214c9bbb3d54b536a2ec92649f9979f929ecd39a0c52f794d63da79b2d3" exitCode=0 Oct 30 18:01:34.843147 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:34.842665 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55b8fd5845-fdl6z" Oct 30 18:01:34.843147 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:34.842700 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55b8fd5845-fdl6z" event={"ID":"6e924469-24ea-4da5-9658-2c35efdea8dd","Type":"ContainerDied","Data":"6c501214c9bbb3d54b536a2ec92649f9979f929ecd39a0c52f794d63da79b2d3"} Oct 30 18:01:34.843147 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:34.842754 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55b8fd5845-fdl6z" event={"ID":"6e924469-24ea-4da5-9658-2c35efdea8dd","Type":"ContainerDied","Data":"cd2758db391d64295e0b3010293afb4f0ce10ee4d5f4cc9de64eeba843540314"} Oct 30 18:01:34.843147 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:34.842777 4455 scope.go:122] "RemoveContainer" containerID="6c501214c9bbb3d54b536a2ec92649f9979f929ecd39a0c52f794d63da79b2d3" Oct 30 18:01:34.854523 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:34.854493 4455 scope.go:122] "RemoveContainer" containerID="6c501214c9bbb3d54b536a2ec92649f9979f929ecd39a0c52f794d63da79b2d3" Oct 30 18:01:34.854808 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:01:34.854790 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c501214c9bbb3d54b536a2ec92649f9979f929ecd39a0c52f794d63da79b2d3\": container with ID starting with 6c501214c9bbb3d54b536a2ec92649f9979f929ecd39a0c52f794d63da79b2d3 not found: ID does not exist" containerID="6c501214c9bbb3d54b536a2ec92649f9979f929ecd39a0c52f794d63da79b2d3" Oct 30 18:01:34.854876 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:34.854817 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c501214c9bbb3d54b536a2ec92649f9979f929ecd39a0c52f794d63da79b2d3"} err="failed to get container status \"6c501214c9bbb3d54b536a2ec92649f9979f929ecd39a0c52f794d63da79b2d3\": rpc error: code = NotFound desc = could not find container \"6c501214c9bbb3d54b536a2ec92649f9979f929ecd39a0c52f794d63da79b2d3\": container with ID starting with 6c501214c9bbb3d54b536a2ec92649f9979f929ecd39a0c52f794d63da79b2d3 not found: ID does not exist" Oct 30 18:01:34.896635 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:34.896595 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-55b8fd5845-fdl6z"] Oct 30 18:01:34.907788 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:34.907755 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-55b8fd5845-fdl6z"] Oct 30 18:01:35.636907 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:01:35.636863 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e924469-24ea-4da5-9658-2c35efdea8dd" path="/var/lib/kubelet/pods/6e924469-24ea-4da5-9658-2c35efdea8dd/volumes" Oct 30 18:02:00.383722 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:02:00.383687 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:02:30.383905 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:02:30.383844 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:03:00.385220 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:03:00.385176 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:03:05.675503 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:05.675468 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a8c61e05-58c947cf67-f4f6f"] Oct 30 18:03:05.676169 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:05.676148 4455 cpu_manager.go:414] "containerMap: removing container" podUID="6e924469-24ea-4da5-9658-2c35efdea8dd" containerName="authorino" Oct 30 18:03:05.676273 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:05.676170 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e924469-24ea-4da5-9658-2c35efdea8dd" containerName="authorino" Oct 30 18:03:05.676389 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:05.676374 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e924469-24ea-4da5-9658-2c35efdea8dd" containerName="authorino" Oct 30 18:03:05.679995 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:05.679975 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a8c61e05-58c947cf67-f4f6f" Oct 30 18:03:05.694143 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:05.694107 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a8c61e05\"" Oct 30 18:03:05.716490 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:05.716461 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a8c61e05-dockercfg-qwvh4\"" Oct 30 18:03:05.720264 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:05.720233 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a8c61e05-58c947cf67-f4f6f"] Oct 30 18:03:05.780153 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:05.780106 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a8c61e05-7f949cc5dd-9n6lk"] Oct 30 18:03:05.781015 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:05.780973 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m87cj\" (UniqueName: \"kubernetes.io/projected/a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f-kube-api-access-m87cj\") pod \"payload-pre-processing-e2e-ait-a8c61e05-58c947cf67-f4f6f\" (UID: \"a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a8c61e05-58c947cf67-f4f6f" Oct 30 18:03:05.781178 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:05.781037 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a8c61e05-58c947cf67-f4f6f\" (UID: \"a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a8c61e05-58c947cf67-f4f6f" Oct 30 18:03:05.787824 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:05.787797 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a8c61e05-7f949cc5dd-9n6lk" Oct 30 18:03:05.809926 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:05.809892 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a8c61e05-7f949cc5dd-9n6lk"] Oct 30 18:03:05.882311 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:05.882277 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52e56ab4-42e3-4f11-87f0-1386b3c15ca2-plugins-config-volume\") pod \"payload-processing-e2e-ait-a8c61e05-7f949cc5dd-9n6lk\" (UID: \"52e56ab4-42e3-4f11-87f0-1386b3c15ca2\") " pod="openshift-ingress/payload-processing-e2e-ait-a8c61e05-7f949cc5dd-9n6lk" Oct 30 18:03:05.882531 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:05.882415 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m87cj\" (UniqueName: \"kubernetes.io/projected/a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f-kube-api-access-m87cj\") pod \"payload-pre-processing-e2e-ait-a8c61e05-58c947cf67-f4f6f\" (UID: \"a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a8c61e05-58c947cf67-f4f6f" Oct 30 18:03:05.882531 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:05.882445 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m9dm\" (UniqueName: \"kubernetes.io/projected/52e56ab4-42e3-4f11-87f0-1386b3c15ca2-kube-api-access-2m9dm\") pod \"payload-processing-e2e-ait-a8c61e05-7f949cc5dd-9n6lk\" (UID: \"52e56ab4-42e3-4f11-87f0-1386b3c15ca2\") " pod="openshift-ingress/payload-processing-e2e-ait-a8c61e05-7f949cc5dd-9n6lk" Oct 30 18:03:05.882531 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:05.882467 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a8c61e05-58c947cf67-f4f6f\" (UID: \"a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a8c61e05-58c947cf67-f4f6f" Oct 30 18:03:05.883221 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:05.883186 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a8c61e05-58c947cf67-f4f6f\" (UID: \"a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a8c61e05-58c947cf67-f4f6f" Oct 30 18:03:05.918269 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:05.918238 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m87cj\" (UniqueName: \"kubernetes.io/projected/a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f-kube-api-access-m87cj\") pod \"payload-pre-processing-e2e-ait-a8c61e05-58c947cf67-f4f6f\" (UID: \"a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a8c61e05-58c947cf67-f4f6f" Oct 30 18:03:05.983733 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:05.983689 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52e56ab4-42e3-4f11-87f0-1386b3c15ca2-plugins-config-volume\") pod \"payload-processing-e2e-ait-a8c61e05-7f949cc5dd-9n6lk\" (UID: \"52e56ab4-42e3-4f11-87f0-1386b3c15ca2\") " pod="openshift-ingress/payload-processing-e2e-ait-a8c61e05-7f949cc5dd-9n6lk" Oct 30 18:03:05.983951 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:05.983812 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m9dm\" (UniqueName: \"kubernetes.io/projected/52e56ab4-42e3-4f11-87f0-1386b3c15ca2-kube-api-access-2m9dm\") pod \"payload-processing-e2e-ait-a8c61e05-7f949cc5dd-9n6lk\" (UID: \"52e56ab4-42e3-4f11-87f0-1386b3c15ca2\") " pod="openshift-ingress/payload-processing-e2e-ait-a8c61e05-7f949cc5dd-9n6lk" Oct 30 18:03:05.984458 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:05.984433 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52e56ab4-42e3-4f11-87f0-1386b3c15ca2-plugins-config-volume\") pod \"payload-processing-e2e-ait-a8c61e05-7f949cc5dd-9n6lk\" (UID: \"52e56ab4-42e3-4f11-87f0-1386b3c15ca2\") " pod="openshift-ingress/payload-processing-e2e-ait-a8c61e05-7f949cc5dd-9n6lk" Oct 30 18:03:05.994509 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:05.994480 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a8c61e05-58c947cf67-f4f6f" Oct 30 18:03:06.027245 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:06.027222 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m9dm\" (UniqueName: \"kubernetes.io/projected/52e56ab4-42e3-4f11-87f0-1386b3c15ca2-kube-api-access-2m9dm\") pod \"payload-processing-e2e-ait-a8c61e05-7f949cc5dd-9n6lk\" (UID: \"52e56ab4-42e3-4f11-87f0-1386b3c15ca2\") " pod="openshift-ingress/payload-processing-e2e-ait-a8c61e05-7f949cc5dd-9n6lk" Oct 30 18:03:06.113404 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:06.113360 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a8c61e05-7f949cc5dd-9n6lk" Oct 30 18:03:06.294409 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:06.294380 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a8c61e05-58c947cf67-f4f6f"] Oct 30 18:03:06.296805 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:03:06.296778 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6d0ea3f_0b58_48e1_ab6c_6b45871fbd9f.slice/crio-299789d32a179a20b2743fd34c9f4dbf197a189c6936baa62ff9c1ed40080cd9 WatchSource:0}: Error finding container 299789d32a179a20b2743fd34c9f4dbf197a189c6936baa62ff9c1ed40080cd9: Status 404 returned error can't find the container with id 299789d32a179a20b2743fd34c9f4dbf197a189c6936baa62ff9c1ed40080cd9 Oct 30 18:03:06.333606 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:06.333576 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a8c61e05-58c947cf67-f4f6f" event={"ID":"a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f","Type":"ContainerStarted","Data":"299789d32a179a20b2743fd34c9f4dbf197a189c6936baa62ff9c1ed40080cd9"} Oct 30 18:03:06.414364 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:06.414178 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a8c61e05-7f949cc5dd-9n6lk"] Oct 30 18:03:06.416336 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:03:06.416301 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52e56ab4_42e3_4f11_87f0_1386b3c15ca2.slice/crio-60cc8def0c9744c3b6e569d06be3030e72968d51f50a83d1e95da3a0e8a7106a WatchSource:0}: Error finding container 60cc8def0c9744c3b6e569d06be3030e72968d51f50a83d1e95da3a0e8a7106a: Status 404 returned error can't find the container with id 60cc8def0c9744c3b6e569d06be3030e72968d51f50a83d1e95da3a0e8a7106a Oct 30 18:03:07.191918 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.191873 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d"] Oct 30 18:03:07.196373 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.196353 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d" Oct 30 18:03:07.230619 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.230588 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a8c61e05-serving-cert\"" Oct 30 18:03:07.230794 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.230717 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 18:03:07.230794 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.230781 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8pmq2\"" Oct 30 18:03:07.234544 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.234517 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d"] Oct 30 18:03:07.297840 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.297810 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcp2n\" (UniqueName: \"kubernetes.io/projected/abdab22c-d4d9-4673-af5e-fc3388d68fa2-kube-api-access-gcp2n\") pod \"maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d\" (UID: \"abdab22c-d4d9-4673-af5e-fc3388d68fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d" Oct 30 18:03:07.298009 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.297862 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abdab22c-d4d9-4673-af5e-fc3388d68fa2-maas-api-tls\") pod \"maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d\" (UID: \"abdab22c-d4d9-4673-af5e-fc3388d68fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d" Oct 30 18:03:07.298009 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.297955 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abdab22c-d4d9-4673-af5e-fc3388d68fa2-openshift-service-ca\") pod \"maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d\" (UID: \"abdab22c-d4d9-4673-af5e-fc3388d68fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d" Oct 30 18:03:07.298009 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.297993 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abdab22c-d4d9-4673-af5e-fc3388d68fa2-metrics-certs\") pod \"maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d\" (UID: \"abdab22c-d4d9-4673-af5e-fc3388d68fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d" Oct 30 18:03:07.356545 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.356498 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a8c61e05-58c947cf67-f4f6f" event={"ID":"a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f","Type":"ContainerStarted","Data":"ebef9631556dfadb115ff8248106b077f2178292c52151924ff1a118c6b24a02"} Oct 30 18:03:07.356727 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.356701 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a8c61e05-58c947cf67-f4f6f" Oct 30 18:03:07.358414 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.358388 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a8c61e05-7f949cc5dd-9n6lk" event={"ID":"52e56ab4-42e3-4f11-87f0-1386b3c15ca2","Type":"ContainerStarted","Data":"73e1d1015056eeb0c4b6f1d90eb9b8fb8bdab9e303c976f9878abb09bb01c860"} Oct 30 18:03:07.358523 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.358437 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a8c61e05-7f949cc5dd-9n6lk" event={"ID":"52e56ab4-42e3-4f11-87f0-1386b3c15ca2","Type":"ContainerStarted","Data":"60cc8def0c9744c3b6e569d06be3030e72968d51f50a83d1e95da3a0e8a7106a"} Oct 30 18:03:07.358523 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.358460 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a8c61e05-7f949cc5dd-9n6lk" Oct 30 18:03:07.399006 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.398977 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abdab22c-d4d9-4673-af5e-fc3388d68fa2-metrics-certs\") pod \"maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d\" (UID: \"abdab22c-d4d9-4673-af5e-fc3388d68fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d" Oct 30 18:03:07.399152 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.399072 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcp2n\" (UniqueName: \"kubernetes.io/projected/abdab22c-d4d9-4673-af5e-fc3388d68fa2-kube-api-access-gcp2n\") pod \"maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d\" (UID: \"abdab22c-d4d9-4673-af5e-fc3388d68fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d" Oct 30 18:03:07.399152 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.399108 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abdab22c-d4d9-4673-af5e-fc3388d68fa2-maas-api-tls\") pod \"maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d\" (UID: \"abdab22c-d4d9-4673-af5e-fc3388d68fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d" Oct 30 18:03:07.399152 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.399145 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abdab22c-d4d9-4673-af5e-fc3388d68fa2-openshift-service-ca\") pod \"maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d\" (UID: \"abdab22c-d4d9-4673-af5e-fc3388d68fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d" Oct 30 18:03:07.399746 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.399724 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abdab22c-d4d9-4673-af5e-fc3388d68fa2-openshift-service-ca\") pod \"maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d\" (UID: \"abdab22c-d4d9-4673-af5e-fc3388d68fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d" Oct 30 18:03:07.401697 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.401646 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a8c61e05-58c947cf67-f4f6f" podStartSLOduration=2.4016325419999998 podStartE2EDuration="2.401632542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:07.400967832 +0000 UTC m=+639.888604866" watchObservedRunningTime="2026-10-30 18:03:07.401632542 +0000 UTC m=+639.889269575" Oct 30 18:03:07.401830 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.401718 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abdab22c-d4d9-4673-af5e-fc3388d68fa2-maas-api-tls\") pod \"maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d\" (UID: \"abdab22c-d4d9-4673-af5e-fc3388d68fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d" Oct 30 18:03:07.401830 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.401768 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abdab22c-d4d9-4673-af5e-fc3388d68fa2-metrics-certs\") pod \"maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d\" (UID: \"abdab22c-d4d9-4673-af5e-fc3388d68fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d" Oct 30 18:03:07.425122 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.425096 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcp2n\" (UniqueName: \"kubernetes.io/projected/abdab22c-d4d9-4673-af5e-fc3388d68fa2-kube-api-access-gcp2n\") pod \"maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d\" (UID: \"abdab22c-d4d9-4673-af5e-fc3388d68fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d" Oct 30 18:03:07.444983 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.444901 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a8c61e05-7f949cc5dd-9n6lk" podStartSLOduration=2.444888334 podStartE2EDuration="2.444888334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:07.444460247 +0000 UTC m=+639.932097281" watchObservedRunningTime="2026-10-30 18:03:07.444888334 +0000 UTC m=+639.932525407" Oct 30 18:03:07.511109 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.511069 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d" Oct 30 18:03:07.579000 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.578963 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ce05c7ef-59894df599-q2kd8"] Oct 30 18:03:07.587345 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.587303 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce05c7ef-59894df599-q2kd8" Oct 30 18:03:07.604066 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.604036 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ce05c7ef-59894df599-q2kd8"] Oct 30 18:03:07.604528 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.604499 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ce05c7ef\"" Oct 30 18:03:07.605108 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.604671 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ce05c7ef-dockercfg-vm29z\"" Oct 30 18:03:07.674860 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.672206 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ce05c7ef-857cf887b6-8knn4"] Oct 30 18:03:07.682680 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.682653 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ce05c7ef-857cf887b6-8knn4" Oct 30 18:03:07.700662 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.700597 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ce05c7ef-857cf887b6-8knn4"] Oct 30 18:03:07.702361 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.702301 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a51e90d-5cff-414d-baca-15fd155cea22-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ce05c7ef-59894df599-q2kd8\" (UID: \"6a51e90d-5cff-414d-baca-15fd155cea22\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ce05c7ef-59894df599-q2kd8" Oct 30 18:03:07.702502 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.702483 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmp59\" (UniqueName: \"kubernetes.io/projected/6a51e90d-5cff-414d-baca-15fd155cea22-kube-api-access-pmp59\") pod \"payload-pre-processing-e2e-ait-ce05c7ef-59894df599-q2kd8\" (UID: \"6a51e90d-5cff-414d-baca-15fd155cea22\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ce05c7ef-59894df599-q2kd8" Oct 30 18:03:07.803229 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.803191 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a51e90d-5cff-414d-baca-15fd155cea22-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ce05c7ef-59894df599-q2kd8\" (UID: \"6a51e90d-5cff-414d-baca-15fd155cea22\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ce05c7ef-59894df599-q2kd8" Oct 30 18:03:07.803603 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.803575 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6lpb\" (UniqueName: \"kubernetes.io/projected/ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b-kube-api-access-d6lpb\") pod \"payload-processing-e2e-ait-ce05c7ef-857cf887b6-8knn4\" (UID: \"ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b\") " pod="openshift-ingress/payload-processing-e2e-ait-ce05c7ef-857cf887b6-8knn4" Oct 30 18:03:07.803824 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.803800 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b-plugins-config-volume\") pod \"payload-processing-e2e-ait-ce05c7ef-857cf887b6-8knn4\" (UID: \"ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b\") " pod="openshift-ingress/payload-processing-e2e-ait-ce05c7ef-857cf887b6-8knn4" Oct 30 18:03:07.803933 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.803915 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmp59\" (UniqueName: \"kubernetes.io/projected/6a51e90d-5cff-414d-baca-15fd155cea22-kube-api-access-pmp59\") pod \"payload-pre-processing-e2e-ait-ce05c7ef-59894df599-q2kd8\" (UID: \"6a51e90d-5cff-414d-baca-15fd155cea22\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ce05c7ef-59894df599-q2kd8" Oct 30 18:03:07.804100 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.804077 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a51e90d-5cff-414d-baca-15fd155cea22-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ce05c7ef-59894df599-q2kd8\" (UID: \"6a51e90d-5cff-414d-baca-15fd155cea22\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ce05c7ef-59894df599-q2kd8" Oct 30 18:03:07.810755 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.810730 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d"] Oct 30 18:03:07.812720 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:03:07.812684 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabdab22c_d4d9_4673_af5e_fc3388d68fa2.slice/crio-85746681d2cc818f02483521fc555d70b9230d066258e4beb8e55a11601ce203 WatchSource:0}: Error finding container 85746681d2cc818f02483521fc555d70b9230d066258e4beb8e55a11601ce203: Status 404 returned error can't find the container with id 85746681d2cc818f02483521fc555d70b9230d066258e4beb8e55a11601ce203 Oct 30 18:03:07.814278 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.814261 4455 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:03:07.827266 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.827240 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmp59\" (UniqueName: \"kubernetes.io/projected/6a51e90d-5cff-414d-baca-15fd155cea22-kube-api-access-pmp59\") pod \"payload-pre-processing-e2e-ait-ce05c7ef-59894df599-q2kd8\" (UID: \"6a51e90d-5cff-414d-baca-15fd155cea22\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ce05c7ef-59894df599-q2kd8" Oct 30 18:03:07.904677 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.904640 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6lpb\" (UniqueName: \"kubernetes.io/projected/ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b-kube-api-access-d6lpb\") pod \"payload-processing-e2e-ait-ce05c7ef-857cf887b6-8knn4\" (UID: \"ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b\") " pod="openshift-ingress/payload-processing-e2e-ait-ce05c7ef-857cf887b6-8knn4" Oct 30 18:03:07.904869 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.904689 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b-plugins-config-volume\") pod \"payload-processing-e2e-ait-ce05c7ef-857cf887b6-8knn4\" (UID: \"ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b\") " pod="openshift-ingress/payload-processing-e2e-ait-ce05c7ef-857cf887b6-8knn4" Oct 30 18:03:07.905603 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.905575 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b-plugins-config-volume\") pod \"payload-processing-e2e-ait-ce05c7ef-857cf887b6-8knn4\" (UID: \"ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b\") " pod="openshift-ingress/payload-processing-e2e-ait-ce05c7ef-857cf887b6-8knn4" Oct 30 18:03:07.907536 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.907515 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce05c7ef-59894df599-q2kd8" Oct 30 18:03:07.927344 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:07.927295 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6lpb\" (UniqueName: \"kubernetes.io/projected/ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b-kube-api-access-d6lpb\") pod \"payload-processing-e2e-ait-ce05c7ef-857cf887b6-8knn4\" (UID: \"ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b\") " pod="openshift-ingress/payload-processing-e2e-ait-ce05c7ef-857cf887b6-8knn4" Oct 30 18:03:08.001521 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:08.001491 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ce05c7ef-857cf887b6-8knn4" Oct 30 18:03:08.212125 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:08.212096 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ce05c7ef-59894df599-q2kd8"] Oct 30 18:03:08.214877 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:03:08.214838 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a51e90d_5cff_414d_baca_15fd155cea22.slice/crio-ec1c08b8b7b853c4b5fc20316c4d9e8c3b0434ecc4cc1770d79e538b59a7b1ae WatchSource:0}: Error finding container ec1c08b8b7b853c4b5fc20316c4d9e8c3b0434ecc4cc1770d79e538b59a7b1ae: Status 404 returned error can't find the container with id ec1c08b8b7b853c4b5fc20316c4d9e8c3b0434ecc4cc1770d79e538b59a7b1ae Oct 30 18:03:08.318887 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:08.316173 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ce05c7ef-857cf887b6-8knn4"] Oct 30 18:03:08.366101 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:08.366071 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d" event={"ID":"abdab22c-d4d9-4673-af5e-fc3388d68fa2","Type":"ContainerStarted","Data":"f0e3a178a01ba2f82e89a148e2351926b9d8b177067d306d43a0c391f1b52cee"} Oct 30 18:03:08.366274 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:08.366250 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d" Oct 30 18:03:08.366274 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:08.366302 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d" event={"ID":"abdab22c-d4d9-4673-af5e-fc3388d68fa2","Type":"ContainerStarted","Data":"85746681d2cc818f02483521fc555d70b9230d066258e4beb8e55a11601ce203"} Oct 30 18:03:08.368278 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:08.368254 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce05c7ef-59894df599-q2kd8" event={"ID":"6a51e90d-5cff-414d-baca-15fd155cea22","Type":"ContainerStarted","Data":"2b9bbabd16468f1bab4dbb674da21a50c1a935d70422faf6a52c9ba18b2e9856"} Oct 30 18:03:08.368423 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:08.368291 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce05c7ef-59894df599-q2kd8" event={"ID":"6a51e90d-5cff-414d-baca-15fd155cea22","Type":"ContainerStarted","Data":"ec1c08b8b7b853c4b5fc20316c4d9e8c3b0434ecc4cc1770d79e538b59a7b1ae"} Oct 30 18:03:08.368731 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:08.368717 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce05c7ef-59894df599-q2kd8" Oct 30 18:03:08.370365 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:08.370298 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ce05c7ef-857cf887b6-8knn4" event={"ID":"ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b","Type":"ContainerStarted","Data":"6e2be04640a0700ae1c2c0863280efcf8c7be602a893e2c8b79d5e163731ad36"} Oct 30 18:03:08.413898 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:08.413843 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d" podStartSLOduration=0.970544109 podStartE2EDuration="1.413823669s" totalImagesPullingTime="443.27956ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:08.405936168 +0000 UTC m=+640.893573203" watchObservedRunningTime="2026-10-30 18:03:08.413823669 +0000 UTC m=+640.901460704" Oct 30 18:03:08.448815 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:08.448762 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce05c7ef-59894df599-q2kd8" podStartSLOduration=1.448749432 podStartE2EDuration="1.448749432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:08.447755732 +0000 UTC m=+640.935392748" watchObservedRunningTime="2026-10-30 18:03:08.448749432 +0000 UTC m=+640.936386466" Oct 30 18:03:08.472622 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:08.472549 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fd500666-77cb9b7459-wfnmq"] Oct 30 18:03:08.476413 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:08.476395 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fd500666-77cb9b7459-wfnmq" Oct 30 18:03:08.491447 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:08.491428 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-fd500666-dockercfg-d7bm7\"" Oct 30 18:03:08.502382 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:08.502362 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-fd500666\"" Oct 30 18:03:08.510480 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:08.510458 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fd500666-77cb9b7459-wfnmq"] Oct 30 18:03:08.579870 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:08.579839 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fd500666-64f88b9c86-s5h2f"] Oct 30 18:03:08.593310 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:08.593267 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fd500666-64f88b9c86-s5h2f" Oct 30 18:03:08.610025 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:08.609069 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fd500666-64f88b9c86-s5h2f"] Oct 30 18:03:08.614581 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:08.614474 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca6dae74-7199-4366-b7cc-c2ca15f2001c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fd500666-77cb9b7459-wfnmq\" (UID: \"ca6dae74-7199-4366-b7cc-c2ca15f2001c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fd500666-77cb9b7459-wfnmq" Oct 30 18:03:08.614663 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:08.614588 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk9cj\" (UniqueName: \"kubernetes.io/projected/ca6dae74-7199-4366-b7cc-c2ca15f2001c-kube-api-access-lk9cj\") pod \"payload-pre-processing-e2e-migrate-fd500666-77cb9b7459-wfnmq\" (UID: \"ca6dae74-7199-4366-b7cc-c2ca15f2001c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fd500666-77cb9b7459-wfnmq" Oct 30 18:03:08.720347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:08.716025 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjfdb\" (UniqueName: \"kubernetes.io/projected/343886fc-aa2f-47aa-93c6-b9b7f4627ce4-kube-api-access-jjfdb\") pod \"payload-processing-e2e-migrate-fd500666-64f88b9c86-s5h2f\" (UID: \"343886fc-aa2f-47aa-93c6-b9b7f4627ce4\") " pod="openshift-ingress/payload-processing-e2e-migrate-fd500666-64f88b9c86-s5h2f" Oct 30 18:03:08.720347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:08.716596 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca6dae74-7199-4366-b7cc-c2ca15f2001c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fd500666-77cb9b7459-wfnmq\" (UID: \"ca6dae74-7199-4366-b7cc-c2ca15f2001c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fd500666-77cb9b7459-wfnmq" Oct 30 18:03:08.720347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:08.716728 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk9cj\" (UniqueName: \"kubernetes.io/projected/ca6dae74-7199-4366-b7cc-c2ca15f2001c-kube-api-access-lk9cj\") pod \"payload-pre-processing-e2e-migrate-fd500666-77cb9b7459-wfnmq\" (UID: \"ca6dae74-7199-4366-b7cc-c2ca15f2001c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fd500666-77cb9b7459-wfnmq" Oct 30 18:03:08.720347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:08.716807 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/343886fc-aa2f-47aa-93c6-b9b7f4627ce4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fd500666-64f88b9c86-s5h2f\" (UID: \"343886fc-aa2f-47aa-93c6-b9b7f4627ce4\") " pod="openshift-ingress/payload-processing-e2e-migrate-fd500666-64f88b9c86-s5h2f" Oct 30 18:03:08.720347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:08.717457 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca6dae74-7199-4366-b7cc-c2ca15f2001c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fd500666-77cb9b7459-wfnmq\" (UID: \"ca6dae74-7199-4366-b7cc-c2ca15f2001c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fd500666-77cb9b7459-wfnmq" Oct 30 18:03:08.743716 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:08.743645 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk9cj\" (UniqueName: \"kubernetes.io/projected/ca6dae74-7199-4366-b7cc-c2ca15f2001c-kube-api-access-lk9cj\") pod \"payload-pre-processing-e2e-migrate-fd500666-77cb9b7459-wfnmq\" (UID: \"ca6dae74-7199-4366-b7cc-c2ca15f2001c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fd500666-77cb9b7459-wfnmq" Oct 30 18:03:08.790526 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:08.790492 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fd500666-77cb9b7459-wfnmq" Oct 30 18:03:08.818164 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:08.818135 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/343886fc-aa2f-47aa-93c6-b9b7f4627ce4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fd500666-64f88b9c86-s5h2f\" (UID: \"343886fc-aa2f-47aa-93c6-b9b7f4627ce4\") " pod="openshift-ingress/payload-processing-e2e-migrate-fd500666-64f88b9c86-s5h2f" Oct 30 18:03:08.818294 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:08.818225 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjfdb\" (UniqueName: \"kubernetes.io/projected/343886fc-aa2f-47aa-93c6-b9b7f4627ce4-kube-api-access-jjfdb\") pod \"payload-processing-e2e-migrate-fd500666-64f88b9c86-s5h2f\" (UID: \"343886fc-aa2f-47aa-93c6-b9b7f4627ce4\") " pod="openshift-ingress/payload-processing-e2e-migrate-fd500666-64f88b9c86-s5h2f" Oct 30 18:03:08.818931 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:08.818909 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/343886fc-aa2f-47aa-93c6-b9b7f4627ce4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fd500666-64f88b9c86-s5h2f\" (UID: \"343886fc-aa2f-47aa-93c6-b9b7f4627ce4\") " pod="openshift-ingress/payload-processing-e2e-migrate-fd500666-64f88b9c86-s5h2f" Oct 30 18:03:08.842748 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:08.842720 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjfdb\" (UniqueName: \"kubernetes.io/projected/343886fc-aa2f-47aa-93c6-b9b7f4627ce4-kube-api-access-jjfdb\") pod \"payload-processing-e2e-migrate-fd500666-64f88b9c86-s5h2f\" (UID: \"343886fc-aa2f-47aa-93c6-b9b7f4627ce4\") " pod="openshift-ingress/payload-processing-e2e-migrate-fd500666-64f88b9c86-s5h2f" Oct 30 18:03:08.924098 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:08.924059 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fd500666-64f88b9c86-s5h2f" Oct 30 18:03:09.091251 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:09.091225 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fd500666-77cb9b7459-wfnmq"] Oct 30 18:03:09.092976 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:03:09.092946 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca6dae74_7199_4366_b7cc_c2ca15f2001c.slice/crio-0af6664cf5836c9f8a800c7a1da1bb061acf7a2342a4ee5a760843d00e7b8a09 WatchSource:0}: Error finding container 0af6664cf5836c9f8a800c7a1da1bb061acf7a2342a4ee5a760843d00e7b8a09: Status 404 returned error can't find the container with id 0af6664cf5836c9f8a800c7a1da1bb061acf7a2342a4ee5a760843d00e7b8a09 Oct 30 18:03:09.234681 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:09.234655 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fd500666-64f88b9c86-s5h2f"] Oct 30 18:03:09.236800 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:03:09.236775 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod343886fc_aa2f_47aa_93c6_b9b7f4627ce4.slice/crio-5ad94e9720fe2a0774f89e558e7cd86a7e1826bc7d192a157415e672e6130364 WatchSource:0}: Error finding container 5ad94e9720fe2a0774f89e558e7cd86a7e1826bc7d192a157415e672e6130364: Status 404 returned error can't find the container with id 5ad94e9720fe2a0774f89e558e7cd86a7e1826bc7d192a157415e672e6130364 Oct 30 18:03:09.378913 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:09.378818 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fd500666-77cb9b7459-wfnmq" event={"ID":"ca6dae74-7199-4366-b7cc-c2ca15f2001c","Type":"ContainerStarted","Data":"381a75684c03a5aac49fcf7d158dbc2985e9fca72083af571549141d237250e2"} Oct 30 18:03:09.379085 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:09.378982 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fd500666-77cb9b7459-wfnmq" Oct 30 18:03:09.379085 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:09.379019 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fd500666-77cb9b7459-wfnmq" event={"ID":"ca6dae74-7199-4366-b7cc-c2ca15f2001c","Type":"ContainerStarted","Data":"0af6664cf5836c9f8a800c7a1da1bb061acf7a2342a4ee5a760843d00e7b8a09"} Oct 30 18:03:09.380931 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:09.380898 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ce05c7ef-857cf887b6-8knn4" event={"ID":"ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b","Type":"ContainerStarted","Data":"c97e0e31dd7438816d7ca218adceb78c8ae3df260360f707dbde7f6c39fc0bb5"} Oct 30 18:03:09.381878 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:09.381850 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ce05c7ef-857cf887b6-8knn4" Oct 30 18:03:09.384253 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:09.384228 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fd500666-64f88b9c86-s5h2f" event={"ID":"343886fc-aa2f-47aa-93c6-b9b7f4627ce4","Type":"ContainerStarted","Data":"0bcf887b5e0b1bab2244b6a0ac8c18bb85ab16a7e2c124f6e97f0f4d690bbfd1"} Oct 30 18:03:09.384385 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:09.384267 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-fd500666-64f88b9c86-s5h2f" Oct 30 18:03:09.384385 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:09.384285 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fd500666-64f88b9c86-s5h2f" event={"ID":"343886fc-aa2f-47aa-93c6-b9b7f4627ce4","Type":"ContainerStarted","Data":"5ad94e9720fe2a0774f89e558e7cd86a7e1826bc7d192a157415e672e6130364"} Oct 30 18:03:09.422155 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:09.422086 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fd500666-77cb9b7459-wfnmq" podStartSLOduration=1.422062279 podStartE2EDuration="1.422062279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:09.419774375 +0000 UTC m=+641.907411408" watchObservedRunningTime="2026-10-30 18:03:09.422062279 +0000 UTC m=+641.909699380" Oct 30 18:03:09.461528 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:09.461476 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-fd500666-64f88b9c86-s5h2f" podStartSLOduration=1.461460071 podStartE2EDuration="1.461460071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:09.460162405 +0000 UTC m=+641.947799441" watchObservedRunningTime="2026-10-30 18:03:09.461460071 +0000 UTC m=+641.949097104" Oct 30 18:03:09.506030 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:09.505968 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ce05c7ef-857cf887b6-8knn4" podStartSLOduration=2.5059527900000003 podStartE2EDuration="2.50595279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:09.504006645 +0000 UTC m=+641.991643679" watchObservedRunningTime="2026-10-30 18:03:09.50595279 +0000 UTC m=+641.993589823" Oct 30 18:03:11.159091 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:11.159059 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47"] Oct 30 18:03:11.162984 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:11.162967 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47" Oct 30 18:03:11.180920 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:11.180895 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ce05c7ef-serving-cert\"" Oct 30 18:03:11.209588 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:11.209403 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47"] Oct 30 18:03:11.247932 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:11.247892 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/648a2b61-90ed-4953-9974-d636fc0db13e-metrics-certs\") pod \"maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47\" (UID: \"648a2b61-90ed-4953-9974-d636fc0db13e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47" Oct 30 18:03:11.248089 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:11.247947 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/648a2b61-90ed-4953-9974-d636fc0db13e-openshift-service-ca\") pod \"maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47\" (UID: \"648a2b61-90ed-4953-9974-d636fc0db13e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47" Oct 30 18:03:11.248177 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:11.248147 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/648a2b61-90ed-4953-9974-d636fc0db13e-maas-api-tls\") pod \"maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47\" (UID: \"648a2b61-90ed-4953-9974-d636fc0db13e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47" Oct 30 18:03:11.248254 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:11.248224 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ktt5\" (UniqueName: \"kubernetes.io/projected/648a2b61-90ed-4953-9974-d636fc0db13e-kube-api-access-5ktt5\") pod \"maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47\" (UID: \"648a2b61-90ed-4953-9974-d636fc0db13e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47" Oct 30 18:03:11.349476 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:11.349444 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/648a2b61-90ed-4953-9974-d636fc0db13e-metrics-certs\") pod \"maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47\" (UID: \"648a2b61-90ed-4953-9974-d636fc0db13e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47" Oct 30 18:03:11.349652 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:11.349496 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/648a2b61-90ed-4953-9974-d636fc0db13e-openshift-service-ca\") pod \"maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47\" (UID: \"648a2b61-90ed-4953-9974-d636fc0db13e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47" Oct 30 18:03:11.349652 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:11.349565 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/648a2b61-90ed-4953-9974-d636fc0db13e-maas-api-tls\") pod \"maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47\" (UID: \"648a2b61-90ed-4953-9974-d636fc0db13e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47" Oct 30 18:03:11.349652 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:11.349599 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ktt5\" (UniqueName: \"kubernetes.io/projected/648a2b61-90ed-4953-9974-d636fc0db13e-kube-api-access-5ktt5\") pod \"maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47\" (UID: \"648a2b61-90ed-4953-9974-d636fc0db13e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47" Oct 30 18:03:11.350360 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:11.350314 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/648a2b61-90ed-4953-9974-d636fc0db13e-openshift-service-ca\") pod \"maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47\" (UID: \"648a2b61-90ed-4953-9974-d636fc0db13e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47" Oct 30 18:03:11.352308 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:11.352279 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/648a2b61-90ed-4953-9974-d636fc0db13e-maas-api-tls\") pod \"maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47\" (UID: \"648a2b61-90ed-4953-9974-d636fc0db13e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47" Oct 30 18:03:11.352444 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:11.352303 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/648a2b61-90ed-4953-9974-d636fc0db13e-metrics-certs\") pod \"maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47\" (UID: \"648a2b61-90ed-4953-9974-d636fc0db13e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47" Oct 30 18:03:11.373032 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:11.373001 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ktt5\" (UniqueName: \"kubernetes.io/projected/648a2b61-90ed-4953-9974-d636fc0db13e-kube-api-access-5ktt5\") pod \"maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47\" (UID: \"648a2b61-90ed-4953-9974-d636fc0db13e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47" Oct 30 18:03:11.477849 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:11.477826 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47" Oct 30 18:03:11.785194 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:11.785121 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47"] Oct 30 18:03:11.787560 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:03:11.787532 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod648a2b61_90ed_4953_9974_d636fc0db13e.slice/crio-5a3a7c97eafafeec4eacd273ef61f3a40826a2ea9ba119d0b3d94e7c16323f44 WatchSource:0}: Error finding container 5a3a7c97eafafeec4eacd273ef61f3a40826a2ea9ba119d0b3d94e7c16323f44: Status 404 returned error can't find the container with id 5a3a7c97eafafeec4eacd273ef61f3a40826a2ea9ba119d0b3d94e7c16323f44 Oct 30 18:03:12.406261 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:12.406159 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47" event={"ID":"648a2b61-90ed-4953-9974-d636fc0db13e","Type":"ContainerStarted","Data":"0c522e73eb1f9bc8276b442b4d027fb6a565ede9a1c007c8900623305312cf55"} Oct 30 18:03:12.406261 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:12.406216 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47" Oct 30 18:03:12.406261 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:12.406234 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47" event={"ID":"648a2b61-90ed-4953-9974-d636fc0db13e","Type":"ContainerStarted","Data":"5a3a7c97eafafeec4eacd273ef61f3a40826a2ea9ba119d0b3d94e7c16323f44"} Oct 30 18:03:12.449994 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:12.449939 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47" podStartSLOduration=1.137625717 podStartE2EDuration="1.449911172s" totalImagesPullingTime="312.285455ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:12.447092513 +0000 UTC m=+644.934729548" watchObservedRunningTime="2026-10-30 18:03:12.449911172 +0000 UTC m=+644.937548207" Oct 30 18:03:13.093123 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:13.093086 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp"] Oct 30 18:03:13.097085 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:13.097062 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp" Oct 30 18:03:13.113471 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:13.113438 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-fd500666-serving-cert\"" Oct 30 18:03:13.128036 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:13.128006 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp"] Oct 30 18:03:13.170615 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:13.170579 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28d593a5-d995-4a16-95d1-e2b349333657-maas-api-tls\") pod \"maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp\" (UID: \"28d593a5-d995-4a16-95d1-e2b349333657\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp" Oct 30 18:03:13.170788 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:13.170749 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28d593a5-d995-4a16-95d1-e2b349333657-metrics-certs\") pod \"maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp\" (UID: \"28d593a5-d995-4a16-95d1-e2b349333657\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp" Oct 30 18:03:13.170897 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:13.170876 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28d593a5-d995-4a16-95d1-e2b349333657-openshift-service-ca\") pod \"maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp\" (UID: \"28d593a5-d995-4a16-95d1-e2b349333657\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp" Oct 30 18:03:13.170967 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:13.170916 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwqdf\" (UniqueName: \"kubernetes.io/projected/28d593a5-d995-4a16-95d1-e2b349333657-kube-api-access-zwqdf\") pod \"maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp\" (UID: \"28d593a5-d995-4a16-95d1-e2b349333657\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp" Oct 30 18:03:13.272179 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:13.272147 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28d593a5-d995-4a16-95d1-e2b349333657-openshift-service-ca\") pod \"maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp\" (UID: \"28d593a5-d995-4a16-95d1-e2b349333657\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp" Oct 30 18:03:13.272179 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:13.272183 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwqdf\" (UniqueName: \"kubernetes.io/projected/28d593a5-d995-4a16-95d1-e2b349333657-kube-api-access-zwqdf\") pod \"maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp\" (UID: \"28d593a5-d995-4a16-95d1-e2b349333657\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp" Oct 30 18:03:13.272428 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:13.272250 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28d593a5-d995-4a16-95d1-e2b349333657-maas-api-tls\") pod \"maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp\" (UID: \"28d593a5-d995-4a16-95d1-e2b349333657\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp" Oct 30 18:03:13.272428 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:13.272385 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28d593a5-d995-4a16-95d1-e2b349333657-metrics-certs\") pod \"maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp\" (UID: \"28d593a5-d995-4a16-95d1-e2b349333657\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp" Oct 30 18:03:13.272942 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:13.272916 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28d593a5-d995-4a16-95d1-e2b349333657-openshift-service-ca\") pod \"maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp\" (UID: \"28d593a5-d995-4a16-95d1-e2b349333657\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp" Oct 30 18:03:13.274928 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:13.274907 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28d593a5-d995-4a16-95d1-e2b349333657-maas-api-tls\") pod \"maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp\" (UID: \"28d593a5-d995-4a16-95d1-e2b349333657\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp" Oct 30 18:03:13.275024 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:13.274955 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28d593a5-d995-4a16-95d1-e2b349333657-metrics-certs\") pod \"maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp\" (UID: \"28d593a5-d995-4a16-95d1-e2b349333657\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp" Oct 30 18:03:13.308472 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:13.308439 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwqdf\" (UniqueName: \"kubernetes.io/projected/28d593a5-d995-4a16-95d1-e2b349333657-kube-api-access-zwqdf\") pod \"maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp\" (UID: \"28d593a5-d995-4a16-95d1-e2b349333657\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp" Oct 30 18:03:13.416719 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:13.416637 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp" Oct 30 18:03:13.717037 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:13.717013 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp"] Oct 30 18:03:13.719493 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:03:13.719469 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28d593a5_d995_4a16_95d1_e2b349333657.slice/crio-f72c154e5d0b2e0aba62e701dd53e9244759e8b976dde99801c9dc951b062232 WatchSource:0}: Error finding container f72c154e5d0b2e0aba62e701dd53e9244759e8b976dde99801c9dc951b062232: Status 404 returned error can't find the container with id f72c154e5d0b2e0aba62e701dd53e9244759e8b976dde99801c9dc951b062232 Oct 30 18:03:14.392002 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:14.391929 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d" Oct 30 18:03:14.419663 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:14.419623 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp" event={"ID":"28d593a5-d995-4a16-95d1-e2b349333657","Type":"ContainerStarted","Data":"7ccb35028aab43904c794f338120d1ce9cf7bc20a348a224cddcaaf5f343f15a"} Oct 30 18:03:14.420035 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:14.419690 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp" event={"ID":"28d593a5-d995-4a16-95d1-e2b349333657","Type":"ContainerStarted","Data":"f72c154e5d0b2e0aba62e701dd53e9244759e8b976dde99801c9dc951b062232"} Oct 30 18:03:14.420035 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:14.419721 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp" Oct 30 18:03:14.424035 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:14.424006 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5e0da928-66b7947b4c-8lsn2"] Oct 30 18:03:14.428970 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:14.428949 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5e0da928-66b7947b4c-8lsn2" Oct 30 18:03:14.440626 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:14.440590 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5e0da928-66b7947b4c-8lsn2"] Oct 30 18:03:14.485865 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:14.485833 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-5e0da928\"" Oct 30 18:03:14.486287 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:14.486266 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx5cf\" (UniqueName: \"kubernetes.io/projected/c253589b-8899-46ee-b9db-9ae7e05d0571-kube-api-access-tx5cf\") pod \"payload-pre-processing-e2e-derive-5e0da928-66b7947b4c-8lsn2\" (UID: \"c253589b-8899-46ee-b9db-9ae7e05d0571\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5e0da928-66b7947b4c-8lsn2" Oct 30 18:03:14.487229 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:14.486952 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c253589b-8899-46ee-b9db-9ae7e05d0571-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5e0da928-66b7947b4c-8lsn2\" (UID: \"c253589b-8899-46ee-b9db-9ae7e05d0571\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5e0da928-66b7947b4c-8lsn2" Oct 30 18:03:14.522336 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:14.522283 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5e0da928-7d8b7dddcb-5q6z8"] Oct 30 18:03:14.527254 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:14.527230 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5e0da928-7d8b7dddcb-5q6z8" Oct 30 18:03:14.534361 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:14.533842 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-5e0da928-dockercfg-kx6th\"" Oct 30 18:03:14.546631 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:14.546586 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp" podStartSLOduration=1.217071269 podStartE2EDuration="1.546574202s" totalImagesPullingTime="329.502933ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:14.543707626 +0000 UTC m=+647.031344660" watchObservedRunningTime="2026-10-30 18:03:14.546574202 +0000 UTC m=+647.034211622" Oct 30 18:03:14.550628 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:14.550599 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5e0da928-7d8b7dddcb-5q6z8"] Oct 30 18:03:14.588343 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:14.588291 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx5cf\" (UniqueName: \"kubernetes.io/projected/c253589b-8899-46ee-b9db-9ae7e05d0571-kube-api-access-tx5cf\") pod \"payload-pre-processing-e2e-derive-5e0da928-66b7947b4c-8lsn2\" (UID: \"c253589b-8899-46ee-b9db-9ae7e05d0571\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5e0da928-66b7947b4c-8lsn2" Oct 30 18:03:14.588537 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:14.588393 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4645b3-3e3d-4330-946b-39543b416f48-plugins-config-volume\") pod \"payload-processing-e2e-derive-5e0da928-7d8b7dddcb-5q6z8\" (UID: \"4e4645b3-3e3d-4330-946b-39543b416f48\") " pod="openshift-ingress/payload-processing-e2e-derive-5e0da928-7d8b7dddcb-5q6z8" Oct 30 18:03:14.588606 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:14.588533 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c253589b-8899-46ee-b9db-9ae7e05d0571-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5e0da928-66b7947b4c-8lsn2\" (UID: \"c253589b-8899-46ee-b9db-9ae7e05d0571\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5e0da928-66b7947b4c-8lsn2" Oct 30 18:03:14.588672 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:14.588619 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tp44\" (UniqueName: \"kubernetes.io/projected/4e4645b3-3e3d-4330-946b-39543b416f48-kube-api-access-9tp44\") pod \"payload-processing-e2e-derive-5e0da928-7d8b7dddcb-5q6z8\" (UID: \"4e4645b3-3e3d-4330-946b-39543b416f48\") " pod="openshift-ingress/payload-processing-e2e-derive-5e0da928-7d8b7dddcb-5q6z8" Oct 30 18:03:14.589554 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:14.589525 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c253589b-8899-46ee-b9db-9ae7e05d0571-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5e0da928-66b7947b4c-8lsn2\" (UID: \"c253589b-8899-46ee-b9db-9ae7e05d0571\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5e0da928-66b7947b4c-8lsn2" Oct 30 18:03:14.615833 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:14.615797 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx5cf\" (UniqueName: \"kubernetes.io/projected/c253589b-8899-46ee-b9db-9ae7e05d0571-kube-api-access-tx5cf\") pod \"payload-pre-processing-e2e-derive-5e0da928-66b7947b4c-8lsn2\" (UID: \"c253589b-8899-46ee-b9db-9ae7e05d0571\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5e0da928-66b7947b4c-8lsn2" Oct 30 18:03:14.691148 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:14.689975 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tp44\" (UniqueName: \"kubernetes.io/projected/4e4645b3-3e3d-4330-946b-39543b416f48-kube-api-access-9tp44\") pod \"payload-processing-e2e-derive-5e0da928-7d8b7dddcb-5q6z8\" (UID: \"4e4645b3-3e3d-4330-946b-39543b416f48\") " pod="openshift-ingress/payload-processing-e2e-derive-5e0da928-7d8b7dddcb-5q6z8" Oct 30 18:03:14.691148 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:14.690110 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4645b3-3e3d-4330-946b-39543b416f48-plugins-config-volume\") pod \"payload-processing-e2e-derive-5e0da928-7d8b7dddcb-5q6z8\" (UID: \"4e4645b3-3e3d-4330-946b-39543b416f48\") " pod="openshift-ingress/payload-processing-e2e-derive-5e0da928-7d8b7dddcb-5q6z8" Oct 30 18:03:14.693034 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:14.693001 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4645b3-3e3d-4330-946b-39543b416f48-plugins-config-volume\") pod \"payload-processing-e2e-derive-5e0da928-7d8b7dddcb-5q6z8\" (UID: \"4e4645b3-3e3d-4330-946b-39543b416f48\") " pod="openshift-ingress/payload-processing-e2e-derive-5e0da928-7d8b7dddcb-5q6z8" Oct 30 18:03:14.714298 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:14.714269 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tp44\" (UniqueName: \"kubernetes.io/projected/4e4645b3-3e3d-4330-946b-39543b416f48-kube-api-access-9tp44\") pod \"payload-processing-e2e-derive-5e0da928-7d8b7dddcb-5q6z8\" (UID: \"4e4645b3-3e3d-4330-946b-39543b416f48\") " pod="openshift-ingress/payload-processing-e2e-derive-5e0da928-7d8b7dddcb-5q6z8" Oct 30 18:03:14.742581 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:14.742537 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5e0da928-66b7947b4c-8lsn2" Oct 30 18:03:14.843757 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:14.843718 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5e0da928-7d8b7dddcb-5q6z8" Oct 30 18:03:15.044091 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:15.044061 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5e0da928-66b7947b4c-8lsn2"] Oct 30 18:03:15.045616 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:03:15.045584 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc253589b_8899_46ee_b9db_9ae7e05d0571.slice/crio-5865916019d488844dca0a3f6da96f06f1a08766b8a638c556517e38664f25d7 WatchSource:0}: Error finding container 5865916019d488844dca0a3f6da96f06f1a08766b8a638c556517e38664f25d7: Status 404 returned error can't find the container with id 5865916019d488844dca0a3f6da96f06f1a08766b8a638c556517e38664f25d7 Oct 30 18:03:15.147408 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:15.147387 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5e0da928-7d8b7dddcb-5q6z8"] Oct 30 18:03:15.148951 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:03:15.148928 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4645b3_3e3d_4330_946b_39543b416f48.slice/crio-22c67080261314f1fb28abc407add2e09208deed2157bdf2d76117034f69272f WatchSource:0}: Error finding container 22c67080261314f1fb28abc407add2e09208deed2157bdf2d76117034f69272f: Status 404 returned error can't find the container with id 22c67080261314f1fb28abc407add2e09208deed2157bdf2d76117034f69272f Oct 30 18:03:15.427915 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:15.427816 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5e0da928-7d8b7dddcb-5q6z8" event={"ID":"4e4645b3-3e3d-4330-946b-39543b416f48","Type":"ContainerStarted","Data":"93e6e73b8543fbd54ff7728aa54179db5780cb3acd61119d7d5b493a77c2c68f"} Oct 30 18:03:15.427915 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:15.427888 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-5e0da928-7d8b7dddcb-5q6z8" Oct 30 18:03:15.428379 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:15.427917 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5e0da928-7d8b7dddcb-5q6z8" event={"ID":"4e4645b3-3e3d-4330-946b-39543b416f48","Type":"ContainerStarted","Data":"22c67080261314f1fb28abc407add2e09208deed2157bdf2d76117034f69272f"} Oct 30 18:03:15.429824 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:15.429788 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5e0da928-66b7947b4c-8lsn2" event={"ID":"c253589b-8899-46ee-b9db-9ae7e05d0571","Type":"ContainerStarted","Data":"80b0a03d5036ca521ebf8eee555e5f3c7cc260ff30fb8c43bfec67daec326a28"} Oct 30 18:03:15.429948 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:15.429843 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5e0da928-66b7947b4c-8lsn2" event={"ID":"c253589b-8899-46ee-b9db-9ae7e05d0571","Type":"ContainerStarted","Data":"5865916019d488844dca0a3f6da96f06f1a08766b8a638c556517e38664f25d7"} Oct 30 18:03:15.429948 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:15.429874 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-5e0da928-66b7947b4c-8lsn2" Oct 30 18:03:15.468930 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:15.468868 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-5e0da928-7d8b7dddcb-5q6z8" podStartSLOduration=1.468848486 podStartE2EDuration="1.468848486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:15.465596044 +0000 UTC m=+647.953233078" watchObservedRunningTime="2026-10-30 18:03:15.468848486 +0000 UTC m=+647.956485523" Oct 30 18:03:15.505253 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:15.505197 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-5e0da928-66b7947b4c-8lsn2" podStartSLOduration=1.5051836939999998 podStartE2EDuration="1.505183694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:15.503899915 +0000 UTC m=+647.991536949" watchObservedRunningTime="2026-10-30 18:03:15.505183694 +0000 UTC m=+647.992820728" Oct 30 18:03:18.374218 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:18.374178 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a8c61e05-58c947cf67-f4f6f" Oct 30 18:03:18.374685 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:18.374267 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a8c61e05-7f949cc5dd-9n6lk" Oct 30 18:03:18.419046 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:18.419018 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47" Oct 30 18:03:18.773769 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:18.773734 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c"] Oct 30 18:03:18.779074 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:18.779053 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c" Oct 30 18:03:18.797298 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:18.797263 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-5e0da928-serving-cert\"" Oct 30 18:03:18.802538 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:18.802507 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c"] Oct 30 18:03:18.832438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:18.832391 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlj8s\" (UniqueName: \"kubernetes.io/projected/a887a798-03ec-426a-894a-aeef866f7f26-kube-api-access-qlj8s\") pod \"maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c\" (UID: \"a887a798-03ec-426a-894a-aeef866f7f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c" Oct 30 18:03:18.832627 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:18.832544 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a887a798-03ec-426a-894a-aeef866f7f26-openshift-service-ca\") pod \"maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c\" (UID: \"a887a798-03ec-426a-894a-aeef866f7f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c" Oct 30 18:03:18.832627 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:18.832595 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a887a798-03ec-426a-894a-aeef866f7f26-maas-api-tls\") pod \"maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c\" (UID: \"a887a798-03ec-426a-894a-aeef866f7f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c" Oct 30 18:03:18.832801 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:18.832778 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a887a798-03ec-426a-894a-aeef866f7f26-metrics-certs\") pod \"maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c\" (UID: \"a887a798-03ec-426a-894a-aeef866f7f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c" Oct 30 18:03:18.933766 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:18.933731 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a887a798-03ec-426a-894a-aeef866f7f26-maas-api-tls\") pod \"maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c\" (UID: \"a887a798-03ec-426a-894a-aeef866f7f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c" Oct 30 18:03:18.933965 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:18.933842 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a887a798-03ec-426a-894a-aeef866f7f26-metrics-certs\") pod \"maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c\" (UID: \"a887a798-03ec-426a-894a-aeef866f7f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c" Oct 30 18:03:18.933965 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:18.933880 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlj8s\" (UniqueName: \"kubernetes.io/projected/a887a798-03ec-426a-894a-aeef866f7f26-kube-api-access-qlj8s\") pod \"maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c\" (UID: \"a887a798-03ec-426a-894a-aeef866f7f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c" Oct 30 18:03:18.933965 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:18.933935 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a887a798-03ec-426a-894a-aeef866f7f26-openshift-service-ca\") pod \"maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c\" (UID: \"a887a798-03ec-426a-894a-aeef866f7f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c" Oct 30 18:03:18.934551 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:18.934525 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a887a798-03ec-426a-894a-aeef866f7f26-openshift-service-ca\") pod \"maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c\" (UID: \"a887a798-03ec-426a-894a-aeef866f7f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c" Oct 30 18:03:18.936598 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:18.936574 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a887a798-03ec-426a-894a-aeef866f7f26-metrics-certs\") pod \"maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c\" (UID: \"a887a798-03ec-426a-894a-aeef866f7f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c" Oct 30 18:03:18.936710 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:18.936689 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a887a798-03ec-426a-894a-aeef866f7f26-maas-api-tls\") pod \"maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c\" (UID: \"a887a798-03ec-426a-894a-aeef866f7f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c" Oct 30 18:03:18.957016 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:18.956993 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlj8s\" (UniqueName: \"kubernetes.io/projected/a887a798-03ec-426a-894a-aeef866f7f26-kube-api-access-qlj8s\") pod \"maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c\" (UID: \"a887a798-03ec-426a-894a-aeef866f7f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c" Oct 30 18:03:19.095484 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:19.095402 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c" Oct 30 18:03:19.397761 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:19.397730 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c"] Oct 30 18:03:19.398900 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:03:19.398875 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda887a798_03ec_426a_894a_aeef866f7f26.slice/crio-ea42af0b44f4aa9386dac402b31673d17fca5955b31d9b7f0cd623fc2cc12bc0 WatchSource:0}: Error finding container ea42af0b44f4aa9386dac402b31673d17fca5955b31d9b7f0cd623fc2cc12bc0: Status 404 returned error can't find the container with id ea42af0b44f4aa9386dac402b31673d17fca5955b31d9b7f0cd623fc2cc12bc0 Oct 30 18:03:19.458658 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:19.458622 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c" event={"ID":"a887a798-03ec-426a-894a-aeef866f7f26","Type":"ContainerStarted","Data":"ea42af0b44f4aa9386dac402b31673d17fca5955b31d9b7f0cd623fc2cc12bc0"} Oct 30 18:03:20.393686 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:20.393654 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-fd500666-64f88b9c86-s5h2f" Oct 30 18:03:20.393849 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:20.393739 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce05c7ef-59894df599-q2kd8" Oct 30 18:03:20.394116 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:20.394095 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fd500666-77cb9b7459-wfnmq" Oct 30 18:03:20.437076 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:20.437051 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp" Oct 30 18:03:20.471104 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:20.471057 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c" event={"ID":"a887a798-03ec-426a-894a-aeef866f7f26","Type":"ContainerStarted","Data":"ca5460d010bd79a09e4d3c971f871834433e6ff7592f2452321e50d89f7119b7"} Oct 30 18:03:20.471340 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:20.471252 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c" Oct 30 18:03:20.606391 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:20.602637 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c" podStartSLOduration=2.318007452 podStartE2EDuration="2.602616671s" totalImagesPullingTime="284.609219ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:20.599767862 +0000 UTC m=+653.087404896" watchObservedRunningTime="2026-10-30 18:03:20.602616671 +0000 UTC m=+653.090253707" Oct 30 18:03:21.401499 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:21.401469 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ce05c7ef-857cf887b6-8knn4" Oct 30 18:03:26.441139 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:26.441108 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-5e0da928-66b7947b4c-8lsn2" Oct 30 18:03:26.441552 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:26.441187 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-5e0da928-7d8b7dddcb-5q6z8" Oct 30 18:03:26.497822 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:26.497593 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c" Oct 30 18:03:27.954981 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:27.954948 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a9a40825-84767fb78c-2fk5b"] Oct 30 18:03:27.960164 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:27.960147 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9a40825-84767fb78c-2fk5b" Oct 30 18:03:27.977070 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:27.977041 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a9a40825\"" Oct 30 18:03:27.988796 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:27.988768 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a9a40825-dockercfg-js7sp\"" Oct 30 18:03:27.993811 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:27.993781 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a9a40825-84767fb78c-2fk5b"] Oct 30 18:03:28.040003 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:28.039954 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a9a40825-7f544d65c4-gclds"] Oct 30 18:03:28.044106 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:28.044075 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wv9k\" (UniqueName: \"kubernetes.io/projected/315ffa4a-2726-439c-b387-d5d148c48524-kube-api-access-7wv9k\") pod \"payload-pre-processing-e2e-ait-a9a40825-84767fb78c-2fk5b\" (UID: \"315ffa4a-2726-439c-b387-d5d148c48524\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a9a40825-84767fb78c-2fk5b" Oct 30 18:03:28.044294 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:28.044232 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/315ffa4a-2726-439c-b387-d5d148c48524-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a9a40825-84767fb78c-2fk5b\" (UID: \"315ffa4a-2726-439c-b387-d5d148c48524\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a9a40825-84767fb78c-2fk5b" Oct 30 18:03:28.052897 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:28.052872 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a9a40825-7f544d65c4-gclds" Oct 30 18:03:28.085046 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:28.078763 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a9a40825-7f544d65c4-gclds"] Oct 30 18:03:28.145361 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:28.145304 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f79v\" (UniqueName: \"kubernetes.io/projected/8ca4b9f7-4a87-474c-9469-118a08eb62ca-kube-api-access-6f79v\") pod \"payload-processing-e2e-ait-a9a40825-7f544d65c4-gclds\" (UID: \"8ca4b9f7-4a87-474c-9469-118a08eb62ca\") " pod="openshift-ingress/payload-processing-e2e-ait-a9a40825-7f544d65c4-gclds" Oct 30 18:03:28.145507 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:28.145431 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/315ffa4a-2726-439c-b387-d5d148c48524-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a9a40825-84767fb78c-2fk5b\" (UID: \"315ffa4a-2726-439c-b387-d5d148c48524\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a9a40825-84767fb78c-2fk5b" Oct 30 18:03:28.145507 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:28.145490 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca4b9f7-4a87-474c-9469-118a08eb62ca-plugins-config-volume\") pod \"payload-processing-e2e-ait-a9a40825-7f544d65c4-gclds\" (UID: \"8ca4b9f7-4a87-474c-9469-118a08eb62ca\") " pod="openshift-ingress/payload-processing-e2e-ait-a9a40825-7f544d65c4-gclds" Oct 30 18:03:28.145605 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:28.145570 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wv9k\" (UniqueName: \"kubernetes.io/projected/315ffa4a-2726-439c-b387-d5d148c48524-kube-api-access-7wv9k\") pod \"payload-pre-processing-e2e-ait-a9a40825-84767fb78c-2fk5b\" (UID: \"315ffa4a-2726-439c-b387-d5d148c48524\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a9a40825-84767fb78c-2fk5b" Oct 30 18:03:28.146165 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:28.146146 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/315ffa4a-2726-439c-b387-d5d148c48524-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a9a40825-84767fb78c-2fk5b\" (UID: \"315ffa4a-2726-439c-b387-d5d148c48524\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a9a40825-84767fb78c-2fk5b" Oct 30 18:03:28.168553 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:28.168518 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wv9k\" (UniqueName: \"kubernetes.io/projected/315ffa4a-2726-439c-b387-d5d148c48524-kube-api-access-7wv9k\") pod \"payload-pre-processing-e2e-ait-a9a40825-84767fb78c-2fk5b\" (UID: \"315ffa4a-2726-439c-b387-d5d148c48524\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a9a40825-84767fb78c-2fk5b" Oct 30 18:03:28.246430 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:28.246392 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca4b9f7-4a87-474c-9469-118a08eb62ca-plugins-config-volume\") pod \"payload-processing-e2e-ait-a9a40825-7f544d65c4-gclds\" (UID: \"8ca4b9f7-4a87-474c-9469-118a08eb62ca\") " pod="openshift-ingress/payload-processing-e2e-ait-a9a40825-7f544d65c4-gclds" Oct 30 18:03:28.246607 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:28.246521 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f79v\" (UniqueName: \"kubernetes.io/projected/8ca4b9f7-4a87-474c-9469-118a08eb62ca-kube-api-access-6f79v\") pod \"payload-processing-e2e-ait-a9a40825-7f544d65c4-gclds\" (UID: \"8ca4b9f7-4a87-474c-9469-118a08eb62ca\") " pod="openshift-ingress/payload-processing-e2e-ait-a9a40825-7f544d65c4-gclds" Oct 30 18:03:28.247168 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:28.247150 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca4b9f7-4a87-474c-9469-118a08eb62ca-plugins-config-volume\") pod \"payload-processing-e2e-ait-a9a40825-7f544d65c4-gclds\" (UID: \"8ca4b9f7-4a87-474c-9469-118a08eb62ca\") " pod="openshift-ingress/payload-processing-e2e-ait-a9a40825-7f544d65c4-gclds" Oct 30 18:03:28.269695 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:28.269661 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f79v\" (UniqueName: \"kubernetes.io/projected/8ca4b9f7-4a87-474c-9469-118a08eb62ca-kube-api-access-6f79v\") pod \"payload-processing-e2e-ait-a9a40825-7f544d65c4-gclds\" (UID: \"8ca4b9f7-4a87-474c-9469-118a08eb62ca\") " pod="openshift-ingress/payload-processing-e2e-ait-a9a40825-7f544d65c4-gclds" Oct 30 18:03:28.274375 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:28.274355 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9a40825-84767fb78c-2fk5b" Oct 30 18:03:28.400040 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:28.400007 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a9a40825-7f544d65c4-gclds" Oct 30 18:03:28.582647 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:28.582619 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a9a40825-84767fb78c-2fk5b"] Oct 30 18:03:28.583394 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:03:28.583368 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod315ffa4a_2726_439c_b387_d5d148c48524.slice/crio-d1e47502f7d49582b4005733ad71a4fdfae28bc3d2c127c48394bd10f9dfe14e WatchSource:0}: Error finding container d1e47502f7d49582b4005733ad71a4fdfae28bc3d2c127c48394bd10f9dfe14e: Status 404 returned error can't find the container with id d1e47502f7d49582b4005733ad71a4fdfae28bc3d2c127c48394bd10f9dfe14e Oct 30 18:03:28.702805 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:28.702774 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a9a40825-7f544d65c4-gclds"] Oct 30 18:03:28.704606 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:03:28.704569 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca4b9f7_4a87_474c_9469_118a08eb62ca.slice/crio-9c21a31b351a9c514a008b40069a70af426c2bdd1dcca0e2ae86ff211e40669c WatchSource:0}: Error finding container 9c21a31b351a9c514a008b40069a70af426c2bdd1dcca0e2ae86ff211e40669c: Status 404 returned error can't find the container with id 9c21a31b351a9c514a008b40069a70af426c2bdd1dcca0e2ae86ff211e40669c Oct 30 18:03:29.545375 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:29.545316 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9a40825-84767fb78c-2fk5b" event={"ID":"315ffa4a-2726-439c-b387-d5d148c48524","Type":"ContainerStarted","Data":"cce0bb36e248c1887db96220b555c808f9d90e352767273db8bb2f42a2cce32f"} Oct 30 18:03:29.545790 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:29.545418 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9a40825-84767fb78c-2fk5b" event={"ID":"315ffa4a-2726-439c-b387-d5d148c48524","Type":"ContainerStarted","Data":"d1e47502f7d49582b4005733ad71a4fdfae28bc3d2c127c48394bd10f9dfe14e"} Oct 30 18:03:29.545790 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:29.545554 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9a40825-84767fb78c-2fk5b" Oct 30 18:03:29.547020 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:29.546996 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a9a40825-7f544d65c4-gclds" event={"ID":"8ca4b9f7-4a87-474c-9469-118a08eb62ca","Type":"ContainerStarted","Data":"71cc57a842681896f3ba9411e93798e6ed144e641722420b8398a920db798502"} Oct 30 18:03:29.547137 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:29.547037 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a9a40825-7f544d65c4-gclds" event={"ID":"8ca4b9f7-4a87-474c-9469-118a08eb62ca","Type":"ContainerStarted","Data":"9c21a31b351a9c514a008b40069a70af426c2bdd1dcca0e2ae86ff211e40669c"} Oct 30 18:03:29.547137 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:29.547102 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a9a40825-7f544d65c4-gclds" Oct 30 18:03:29.630154 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:29.630080 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9a40825-84767fb78c-2fk5b" podStartSLOduration=2.630061457 podStartE2EDuration="2.630061457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:29.625845633 +0000 UTC m=+662.113482668" watchObservedRunningTime="2026-10-30 18:03:29.630061457 +0000 UTC m=+662.117698492" Oct 30 18:03:30.384840 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:03:30.384806 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:03:35.325096 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:35.325026 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a9a40825-7f544d65c4-gclds" podStartSLOduration=7.325005867 podStartE2EDuration="7.325005867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:29.690932872 +0000 UTC m=+662.178569905" watchObservedRunningTime="2026-10-30 18:03:35.325005867 +0000 UTC m=+667.812642903" Oct 30 18:03:35.328686 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:35.328663 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fd500666-64f88b9c86-s5h2f"] Oct 30 18:03:35.375648 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:35.375613 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fd500666-77cb9b7459-wfnmq"] Oct 30 18:03:37.317548 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:37.317516 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ce05c7ef-857cf887b6-8knn4"] Oct 30 18:03:37.374702 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:37.374673 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ce05c7ef-59894df599-q2kd8"] Oct 30 18:03:40.329631 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:40.329547 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-fd500666-64f88b9c86-s5h2f" podUID="343886fc-aa2f-47aa-93c6-b9b7f4627ce4" containerName="payload-processing" containerID="cri-o://0bcf887b5e0b1bab2244b6a0ac8c18bb85ab16a7e2c124f6e97f0f4d690bbfd1" gracePeriod=25 Oct 30 18:03:40.376100 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:40.376053 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fd500666-77cb9b7459-wfnmq" podUID="ca6dae74-7199-4366-b7cc-c2ca15f2001c" containerName="payload-pre-processing" containerID="cri-o://381a75684c03a5aac49fcf7d158dbc2985e9fca72083af571549141d237250e2" gracePeriod=25 Oct 30 18:03:40.391187 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:03:40.391164 4455 logging.go:55] [core] [Channel #212 SubChannel #215]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.64:50052", ServerName: "10.134.0.64:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.64:50052: connect: connection refused" Oct 30 18:03:40.391307 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:03:40.391165 4455 logging.go:55] [core] [Channel #213 SubChannel #216]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.63:50052", ServerName: "10.134.0.63:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.63:50052: connect: connection refused" Oct 30 18:03:40.535173 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:40.535139 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a8c61e05-7f949cc5dd-9n6lk"] Oct 30 18:03:40.559235 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:40.558376 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a9a40825-7f544d65c4-gclds" Oct 30 18:03:40.559808 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:40.559568 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9a40825-84767fb78c-2fk5b" Oct 30 18:03:40.577739 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:40.577701 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a8c61e05-58c947cf67-f4f6f"] Oct 30 18:03:40.624199 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:40.624160 4455 generic.go:356] "Generic (PLEG): container finished" podID="343886fc-aa2f-47aa-93c6-b9b7f4627ce4" containerID="0bcf887b5e0b1bab2244b6a0ac8c18bb85ab16a7e2c124f6e97f0f4d690bbfd1" exitCode=0 Oct 30 18:03:40.624497 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:40.624473 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fd500666-64f88b9c86-s5h2f" event={"ID":"343886fc-aa2f-47aa-93c6-b9b7f4627ce4","Type":"ContainerDied","Data":"0bcf887b5e0b1bab2244b6a0ac8c18bb85ab16a7e2c124f6e97f0f4d690bbfd1"} Oct 30 18:03:40.630428 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:40.630270 4455 generic.go:356] "Generic (PLEG): container finished" podID="ca6dae74-7199-4366-b7cc-c2ca15f2001c" containerID="381a75684c03a5aac49fcf7d158dbc2985e9fca72083af571549141d237250e2" exitCode=0 Oct 30 18:03:40.630428 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:40.630375 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fd500666-77cb9b7459-wfnmq" event={"ID":"ca6dae74-7199-4366-b7cc-c2ca15f2001c","Type":"ContainerDied","Data":"381a75684c03a5aac49fcf7d158dbc2985e9fca72083af571549141d237250e2"} Oct 30 18:03:40.696588 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:40.696501 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fd500666-64f88b9c86-s5h2f" Oct 30 18:03:40.741876 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:40.741848 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fd500666-77cb9b7459-wfnmq" Oct 30 18:03:40.791842 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:40.791805 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/343886fc-aa2f-47aa-93c6-b9b7f4627ce4-kube-api-access-jjfdb\" (UniqueName: \"kubernetes.io/projected/343886fc-aa2f-47aa-93c6-b9b7f4627ce4-kube-api-access-jjfdb\") pod \"343886fc-aa2f-47aa-93c6-b9b7f4627ce4\" (UID: \"343886fc-aa2f-47aa-93c6-b9b7f4627ce4\") " Oct 30 18:03:40.792020 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:40.791907 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/343886fc-aa2f-47aa-93c6-b9b7f4627ce4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/343886fc-aa2f-47aa-93c6-b9b7f4627ce4-plugins-config-volume\") pod \"343886fc-aa2f-47aa-93c6-b9b7f4627ce4\" (UID: \"343886fc-aa2f-47aa-93c6-b9b7f4627ce4\") " Oct 30 18:03:40.792403 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:40.792372 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/343886fc-aa2f-47aa-93c6-b9b7f4627ce4-plugins-config-volume" pod "343886fc-aa2f-47aa-93c6-b9b7f4627ce4" (UID: "343886fc-aa2f-47aa-93c6-b9b7f4627ce4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:03:40.794718 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:40.794687 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/343886fc-aa2f-47aa-93c6-b9b7f4627ce4-kube-api-access-jjfdb" pod "343886fc-aa2f-47aa-93c6-b9b7f4627ce4" (UID: "343886fc-aa2f-47aa-93c6-b9b7f4627ce4"). InnerVolumeSpecName "kube-api-access-jjfdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:40.826851 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:40.826817 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47"] Oct 30 18:03:40.827028 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:40.827009 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47" podUID="648a2b61-90ed-4953-9974-d636fc0db13e" containerName="maas-api" containerID="cri-o://0c522e73eb1f9bc8276b442b4d027fb6a565ede9a1c007c8900623305312cf55" gracePeriod=30 Oct 30 18:03:40.893267 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:40.893176 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca6dae74-7199-4366-b7cc-c2ca15f2001c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca6dae74-7199-4366-b7cc-c2ca15f2001c-plugins-config-volume\") pod \"ca6dae74-7199-4366-b7cc-c2ca15f2001c\" (UID: \"ca6dae74-7199-4366-b7cc-c2ca15f2001c\") " Oct 30 18:03:40.893267 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:40.893225 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca6dae74-7199-4366-b7cc-c2ca15f2001c-kube-api-access-lk9cj\" (UniqueName: \"kubernetes.io/projected/ca6dae74-7199-4366-b7cc-c2ca15f2001c-kube-api-access-lk9cj\") pod \"ca6dae74-7199-4366-b7cc-c2ca15f2001c\" (UID: \"ca6dae74-7199-4366-b7cc-c2ca15f2001c\") " Oct 30 18:03:40.893642 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:40.893614 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca6dae74-7199-4366-b7cc-c2ca15f2001c-plugins-config-volume" pod "ca6dae74-7199-4366-b7cc-c2ca15f2001c" (UID: "ca6dae74-7199-4366-b7cc-c2ca15f2001c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:03:40.893719 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:40.893653 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjfdb\" (UniqueName: \"kubernetes.io/projected/343886fc-aa2f-47aa-93c6-b9b7f4627ce4-kube-api-access-jjfdb\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:03:40.893719 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:40.893668 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/343886fc-aa2f-47aa-93c6-b9b7f4627ce4-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:03:40.895694 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:40.895663 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca6dae74-7199-4366-b7cc-c2ca15f2001c-kube-api-access-lk9cj" pod "ca6dae74-7199-4366-b7cc-c2ca15f2001c" (UID: "ca6dae74-7199-4366-b7cc-c2ca15f2001c"). InnerVolumeSpecName "kube-api-access-lk9cj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:40.994633 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:40.994604 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca6dae74-7199-4366-b7cc-c2ca15f2001c-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:03:40.994633 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:40.994630 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk9cj\" (UniqueName: \"kubernetes.io/projected/ca6dae74-7199-4366-b7cc-c2ca15f2001c-kube-api-access-lk9cj\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:03:41.078497 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.078471 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47" Oct 30 18:03:41.196550 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.196469 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/648a2b61-90ed-4953-9974-d636fc0db13e-kube-api-access-5ktt5\" (UniqueName: \"kubernetes.io/projected/648a2b61-90ed-4953-9974-d636fc0db13e-kube-api-access-5ktt5\") pod \"648a2b61-90ed-4953-9974-d636fc0db13e\" (UID: \"648a2b61-90ed-4953-9974-d636fc0db13e\") " Oct 30 18:03:41.196720 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.196594 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/648a2b61-90ed-4953-9974-d636fc0db13e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/648a2b61-90ed-4953-9974-d636fc0db13e-maas-api-tls\") pod \"648a2b61-90ed-4953-9974-d636fc0db13e\" (UID: \"648a2b61-90ed-4953-9974-d636fc0db13e\") " Oct 30 18:03:41.196720 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.196651 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/648a2b61-90ed-4953-9974-d636fc0db13e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/648a2b61-90ed-4953-9974-d636fc0db13e-openshift-service-ca\") pod \"648a2b61-90ed-4953-9974-d636fc0db13e\" (UID: \"648a2b61-90ed-4953-9974-d636fc0db13e\") " Oct 30 18:03:41.196720 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.196692 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/648a2b61-90ed-4953-9974-d636fc0db13e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/648a2b61-90ed-4953-9974-d636fc0db13e-metrics-certs\") pod \"648a2b61-90ed-4953-9974-d636fc0db13e\" (UID: \"648a2b61-90ed-4953-9974-d636fc0db13e\") " Oct 30 18:03:41.197134 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.197099 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/648a2b61-90ed-4953-9974-d636fc0db13e-openshift-service-ca" pod "648a2b61-90ed-4953-9974-d636fc0db13e" (UID: "648a2b61-90ed-4953-9974-d636fc0db13e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:03:41.198972 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.198942 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/648a2b61-90ed-4953-9974-d636fc0db13e-metrics-certs" pod "648a2b61-90ed-4953-9974-d636fc0db13e" (UID: "648a2b61-90ed-4953-9974-d636fc0db13e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:03:41.199478 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.199456 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/648a2b61-90ed-4953-9974-d636fc0db13e-kube-api-access-5ktt5" pod "648a2b61-90ed-4953-9974-d636fc0db13e" (UID: "648a2b61-90ed-4953-9974-d636fc0db13e"). InnerVolumeSpecName "kube-api-access-5ktt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:41.199556 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.199456 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/648a2b61-90ed-4953-9974-d636fc0db13e-maas-api-tls" pod "648a2b61-90ed-4953-9974-d636fc0db13e" (UID: "648a2b61-90ed-4953-9974-d636fc0db13e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:03:41.232960 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.232931 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp"] Oct 30 18:03:41.233137 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.233117 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp" podUID="28d593a5-d995-4a16-95d1-e2b349333657" containerName="maas-api" containerID="cri-o://7ccb35028aab43904c794f338120d1ce9cf7bc20a348a224cddcaaf5f343f15a" gracePeriod=30 Oct 30 18:03:41.298310 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.298277 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/648a2b61-90ed-4953-9974-d636fc0db13e-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:03:41.298310 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.298309 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/648a2b61-90ed-4953-9974-d636fc0db13e-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:03:41.298310 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.298347 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ktt5\" (UniqueName: \"kubernetes.io/projected/648a2b61-90ed-4953-9974-d636fc0db13e-kube-api-access-5ktt5\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:03:41.298559 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.298358 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/648a2b61-90ed-4953-9974-d636fc0db13e-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:03:41.391550 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.391506 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fd500666-77cb9b7459-wfnmq" podUID="ca6dae74-7199-4366-b7cc-c2ca15f2001c" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.63:50052\" within 1s: context deadline exceeded" Oct 30 18:03:41.392052 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.391507 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-migrate-fd500666-64f88b9c86-s5h2f" podUID="343886fc-aa2f-47aa-93c6-b9b7f4627ce4" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.64:50052\" within 1s: context deadline exceeded" Oct 30 18:03:41.529692 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.529652 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp" Oct 30 18:03:41.650444 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.650280 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fd500666-64f88b9c86-s5h2f" Oct 30 18:03:41.650444 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.650415 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fd500666-64f88b9c86-s5h2f" event={"ID":"343886fc-aa2f-47aa-93c6-b9b7f4627ce4","Type":"ContainerDied","Data":"5ad94e9720fe2a0774f89e558e7cd86a7e1826bc7d192a157415e672e6130364"} Oct 30 18:03:41.650606 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.650483 4455 scope.go:122] "RemoveContainer" containerID="0bcf887b5e0b1bab2244b6a0ac8c18bb85ab16a7e2c124f6e97f0f4d690bbfd1" Oct 30 18:03:41.652780 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.652755 4455 generic.go:356] "Generic (PLEG): container finished" podID="648a2b61-90ed-4953-9974-d636fc0db13e" containerID="0c522e73eb1f9bc8276b442b4d027fb6a565ede9a1c007c8900623305312cf55" exitCode=0 Oct 30 18:03:41.652944 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.652923 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47" Oct 30 18:03:41.653033 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.652947 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47" event={"ID":"648a2b61-90ed-4953-9974-d636fc0db13e","Type":"ContainerDied","Data":"0c522e73eb1f9bc8276b442b4d027fb6a565ede9a1c007c8900623305312cf55"} Oct 30 18:03:41.653033 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.652994 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47" event={"ID":"648a2b61-90ed-4953-9974-d636fc0db13e","Type":"ContainerDied","Data":"5a3a7c97eafafeec4eacd273ef61f3a40826a2ea9ba119d0b3d94e7c16323f44"} Oct 30 18:03:41.663249 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.663218 4455 generic.go:356] "Generic (PLEG): container finished" podID="28d593a5-d995-4a16-95d1-e2b349333657" containerID="7ccb35028aab43904c794f338120d1ce9cf7bc20a348a224cddcaaf5f343f15a" exitCode=0 Oct 30 18:03:41.663418 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.663404 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp" Oct 30 18:03:41.663570 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.663550 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp" event={"ID":"28d593a5-d995-4a16-95d1-e2b349333657","Type":"ContainerDied","Data":"7ccb35028aab43904c794f338120d1ce9cf7bc20a348a224cddcaaf5f343f15a"} Oct 30 18:03:41.663633 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.663596 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp" event={"ID":"28d593a5-d995-4a16-95d1-e2b349333657","Type":"ContainerDied","Data":"f72c154e5d0b2e0aba62e701dd53e9244759e8b976dde99801c9dc951b062232"} Oct 30 18:03:41.665265 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.665238 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fd500666-77cb9b7459-wfnmq" event={"ID":"ca6dae74-7199-4366-b7cc-c2ca15f2001c","Type":"ContainerDied","Data":"0af6664cf5836c9f8a800c7a1da1bb061acf7a2342a4ee5a760843d00e7b8a09"} Oct 30 18:03:41.665564 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.665373 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fd500666-77cb9b7459-wfnmq" Oct 30 18:03:41.669076 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.669057 4455 scope.go:122] "RemoveContainer" containerID="0c522e73eb1f9bc8276b442b4d027fb6a565ede9a1c007c8900623305312cf55" Oct 30 18:03:41.679410 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.679392 4455 scope.go:122] "RemoveContainer" containerID="0c522e73eb1f9bc8276b442b4d027fb6a565ede9a1c007c8900623305312cf55" Oct 30 18:03:41.679741 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:03:41.679714 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c522e73eb1f9bc8276b442b4d027fb6a565ede9a1c007c8900623305312cf55\": container with ID starting with 0c522e73eb1f9bc8276b442b4d027fb6a565ede9a1c007c8900623305312cf55 not found: ID does not exist" containerID="0c522e73eb1f9bc8276b442b4d027fb6a565ede9a1c007c8900623305312cf55" Oct 30 18:03:41.679846 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.679741 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c522e73eb1f9bc8276b442b4d027fb6a565ede9a1c007c8900623305312cf55"} err="failed to get container status \"0c522e73eb1f9bc8276b442b4d027fb6a565ede9a1c007c8900623305312cf55\": rpc error: code = NotFound desc = could not find container \"0c522e73eb1f9bc8276b442b4d027fb6a565ede9a1c007c8900623305312cf55\": container with ID starting with 0c522e73eb1f9bc8276b442b4d027fb6a565ede9a1c007c8900623305312cf55 not found: ID does not exist" Oct 30 18:03:41.679846 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.679758 4455 scope.go:122] "RemoveContainer" containerID="7ccb35028aab43904c794f338120d1ce9cf7bc20a348a224cddcaaf5f343f15a" Oct 30 18:03:41.692923 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.692901 4455 scope.go:122] "RemoveContainer" containerID="7ccb35028aab43904c794f338120d1ce9cf7bc20a348a224cddcaaf5f343f15a" Oct 30 18:03:41.693265 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:03:41.693236 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ccb35028aab43904c794f338120d1ce9cf7bc20a348a224cddcaaf5f343f15a\": container with ID starting with 7ccb35028aab43904c794f338120d1ce9cf7bc20a348a224cddcaaf5f343f15a not found: ID does not exist" containerID="7ccb35028aab43904c794f338120d1ce9cf7bc20a348a224cddcaaf5f343f15a" Oct 30 18:03:41.693379 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.693277 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ccb35028aab43904c794f338120d1ce9cf7bc20a348a224cddcaaf5f343f15a"} err="failed to get container status \"7ccb35028aab43904c794f338120d1ce9cf7bc20a348a224cddcaaf5f343f15a\": rpc error: code = NotFound desc = could not find container \"7ccb35028aab43904c794f338120d1ce9cf7bc20a348a224cddcaaf5f343f15a\": container with ID starting with 7ccb35028aab43904c794f338120d1ce9cf7bc20a348a224cddcaaf5f343f15a not found: ID does not exist" Oct 30 18:03:41.693379 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.693300 4455 scope.go:122] "RemoveContainer" containerID="381a75684c03a5aac49fcf7d158dbc2985e9fca72083af571549141d237250e2" Oct 30 18:03:41.705038 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.704570 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28d593a5-d995-4a16-95d1-e2b349333657-metrics-certs\" (UniqueName: \"kubernetes.io/secret/28d593a5-d995-4a16-95d1-e2b349333657-metrics-certs\") pod \"28d593a5-d995-4a16-95d1-e2b349333657\" (UID: \"28d593a5-d995-4a16-95d1-e2b349333657\") " Oct 30 18:03:41.705038 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.704624 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28d593a5-d995-4a16-95d1-e2b349333657-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28d593a5-d995-4a16-95d1-e2b349333657-openshift-service-ca\") pod \"28d593a5-d995-4a16-95d1-e2b349333657\" (UID: \"28d593a5-d995-4a16-95d1-e2b349333657\") " Oct 30 18:03:41.705038 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.704692 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28d593a5-d995-4a16-95d1-e2b349333657-kube-api-access-zwqdf\" (UniqueName: \"kubernetes.io/projected/28d593a5-d995-4a16-95d1-e2b349333657-kube-api-access-zwqdf\") pod \"28d593a5-d995-4a16-95d1-e2b349333657\" (UID: \"28d593a5-d995-4a16-95d1-e2b349333657\") " Oct 30 18:03:41.705038 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.704917 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28d593a5-d995-4a16-95d1-e2b349333657-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28d593a5-d995-4a16-95d1-e2b349333657-maas-api-tls\") pod \"28d593a5-d995-4a16-95d1-e2b349333657\" (UID: \"28d593a5-d995-4a16-95d1-e2b349333657\") " Oct 30 18:03:41.705349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.705220 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28d593a5-d995-4a16-95d1-e2b349333657-openshift-service-ca" pod "28d593a5-d995-4a16-95d1-e2b349333657" (UID: "28d593a5-d995-4a16-95d1-e2b349333657"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:03:41.706348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.705966 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28d593a5-d995-4a16-95d1-e2b349333657-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:03:41.711392 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.710242 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28d593a5-d995-4a16-95d1-e2b349333657-metrics-certs" pod "28d593a5-d995-4a16-95d1-e2b349333657" (UID: "28d593a5-d995-4a16-95d1-e2b349333657"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:03:41.715645 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.715619 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28d593a5-d995-4a16-95d1-e2b349333657-kube-api-access-zwqdf" pod "28d593a5-d995-4a16-95d1-e2b349333657" (UID: "28d593a5-d995-4a16-95d1-e2b349333657"). InnerVolumeSpecName "kube-api-access-zwqdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:41.715645 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.715632 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28d593a5-d995-4a16-95d1-e2b349333657-maas-api-tls" pod "28d593a5-d995-4a16-95d1-e2b349333657" (UID: "28d593a5-d995-4a16-95d1-e2b349333657"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:03:41.727654 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.727630 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fd500666-64f88b9c86-s5h2f"] Oct 30 18:03:41.735016 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.734994 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fd500666-64f88b9c86-s5h2f"] Oct 30 18:03:41.787312 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.787232 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d"] Oct 30 18:03:41.787606 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.787575 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d" podUID="abdab22c-d4d9-4673-af5e-fc3388d68fa2" containerName="maas-api" containerID="cri-o://f0e3a178a01ba2f82e89a148e2351926b9d8b177067d306d43a0c391f1b52cee" gracePeriod=30 Oct 30 18:03:41.807209 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.807184 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28d593a5-d995-4a16-95d1-e2b349333657-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:03:41.807301 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.807211 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwqdf\" (UniqueName: \"kubernetes.io/projected/28d593a5-d995-4a16-95d1-e2b349333657-kube-api-access-zwqdf\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:03:41.807301 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.807222 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28d593a5-d995-4a16-95d1-e2b349333657-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:03:41.818463 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.818433 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47"] Oct 30 18:03:41.835212 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.835178 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ce05c7ef-66c78f8fb-4tm47"] Oct 30 18:03:41.933723 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.933684 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fd500666-77cb9b7459-wfnmq"] Oct 30 18:03:41.936822 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:41.936781 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fd500666-77cb9b7459-wfnmq"] Oct 30 18:03:42.038487 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.038422 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp"] Oct 30 18:03:42.040492 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.040470 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d" Oct 30 18:03:42.057756 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.057732 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fd500666-57f99fd8f4-vp9pp"] Oct 30 18:03:42.212001 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.211966 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abdab22c-d4d9-4673-af5e-fc3388d68fa2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abdab22c-d4d9-4673-af5e-fc3388d68fa2-maas-api-tls\") pod \"abdab22c-d4d9-4673-af5e-fc3388d68fa2\" (UID: \"abdab22c-d4d9-4673-af5e-fc3388d68fa2\") " Oct 30 18:03:42.212165 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.212086 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abdab22c-d4d9-4673-af5e-fc3388d68fa2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/abdab22c-d4d9-4673-af5e-fc3388d68fa2-metrics-certs\") pod \"abdab22c-d4d9-4673-af5e-fc3388d68fa2\" (UID: \"abdab22c-d4d9-4673-af5e-fc3388d68fa2\") " Oct 30 18:03:42.212165 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.212145 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abdab22c-d4d9-4673-af5e-fc3388d68fa2-kube-api-access-gcp2n\" (UniqueName: \"kubernetes.io/projected/abdab22c-d4d9-4673-af5e-fc3388d68fa2-kube-api-access-gcp2n\") pod \"abdab22c-d4d9-4673-af5e-fc3388d68fa2\" (UID: \"abdab22c-d4d9-4673-af5e-fc3388d68fa2\") " Oct 30 18:03:42.212306 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.212178 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abdab22c-d4d9-4673-af5e-fc3388d68fa2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abdab22c-d4d9-4673-af5e-fc3388d68fa2-openshift-service-ca\") pod \"abdab22c-d4d9-4673-af5e-fc3388d68fa2\" (UID: \"abdab22c-d4d9-4673-af5e-fc3388d68fa2\") " Oct 30 18:03:42.212688 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.212658 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abdab22c-d4d9-4673-af5e-fc3388d68fa2-openshift-service-ca" pod "abdab22c-d4d9-4673-af5e-fc3388d68fa2" (UID: "abdab22c-d4d9-4673-af5e-fc3388d68fa2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:03:42.214463 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.214439 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abdab22c-d4d9-4673-af5e-fc3388d68fa2-metrics-certs" pod "abdab22c-d4d9-4673-af5e-fc3388d68fa2" (UID: "abdab22c-d4d9-4673-af5e-fc3388d68fa2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:03:42.214564 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.214476 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abdab22c-d4d9-4673-af5e-fc3388d68fa2-kube-api-access-gcp2n" pod "abdab22c-d4d9-4673-af5e-fc3388d68fa2" (UID: "abdab22c-d4d9-4673-af5e-fc3388d68fa2"). InnerVolumeSpecName "kube-api-access-gcp2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:42.214682 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.214666 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abdab22c-d4d9-4673-af5e-fc3388d68fa2-maas-api-tls" pod "abdab22c-d4d9-4673-af5e-fc3388d68fa2" (UID: "abdab22c-d4d9-4673-af5e-fc3388d68fa2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:03:42.313810 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.313723 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abdab22c-d4d9-4673-af5e-fc3388d68fa2-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:03:42.313810 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.313762 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abdab22c-d4d9-4673-af5e-fc3388d68fa2-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:03:42.313810 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.313777 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcp2n\" (UniqueName: \"kubernetes.io/projected/abdab22c-d4d9-4673-af5e-fc3388d68fa2-kube-api-access-gcp2n\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:03:42.313810 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.313793 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abdab22c-d4d9-4673-af5e-fc3388d68fa2-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:03:42.318225 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.318189 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ce05c7ef-857cf887b6-8knn4" podUID="ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b" containerName="payload-processing" containerID="cri-o://c97e0e31dd7438816d7ca218adceb78c8ae3df260360f707dbde7f6c39fc0bb5" gracePeriod=25 Oct 30 18:03:42.375697 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.375651 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce05c7ef-59894df599-q2kd8" podUID="6a51e90d-5cff-414d-baca-15fd155cea22" containerName="payload-pre-processing" containerID="cri-o://2b9bbabd16468f1bab4dbb674da21a50c1a935d70422faf6a52c9ba18b2e9856" gracePeriod=25 Oct 30 18:03:42.649223 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.649198 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ce05c7ef-857cf887b6-8knn4" Oct 30 18:03:42.662970 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.662939 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5e0da928-7d8b7dddcb-5q6z8"] Oct 30 18:03:42.677312 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.677284 4455 generic.go:356] "Generic (PLEG): container finished" podID="abdab22c-d4d9-4673-af5e-fc3388d68fa2" containerID="f0e3a178a01ba2f82e89a148e2351926b9d8b177067d306d43a0c391f1b52cee" exitCode=0 Oct 30 18:03:42.677469 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.677381 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d" Oct 30 18:03:42.677520 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.677367 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d" event={"ID":"abdab22c-d4d9-4673-af5e-fc3388d68fa2","Type":"ContainerDied","Data":"f0e3a178a01ba2f82e89a148e2351926b9d8b177067d306d43a0c391f1b52cee"} Oct 30 18:03:42.677575 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.677545 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d" event={"ID":"abdab22c-d4d9-4673-af5e-fc3388d68fa2","Type":"ContainerDied","Data":"85746681d2cc818f02483521fc555d70b9230d066258e4beb8e55a11601ce203"} Oct 30 18:03:42.677620 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.677583 4455 scope.go:122] "RemoveContainer" containerID="f0e3a178a01ba2f82e89a148e2351926b9d8b177067d306d43a0c391f1b52cee" Oct 30 18:03:42.681459 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.681429 4455 generic.go:356] "Generic (PLEG): container finished" podID="6a51e90d-5cff-414d-baca-15fd155cea22" containerID="2b9bbabd16468f1bab4dbb674da21a50c1a935d70422faf6a52c9ba18b2e9856" exitCode=0 Oct 30 18:03:42.681599 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.681493 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce05c7ef-59894df599-q2kd8" event={"ID":"6a51e90d-5cff-414d-baca-15fd155cea22","Type":"ContainerDied","Data":"2b9bbabd16468f1bab4dbb674da21a50c1a935d70422faf6a52c9ba18b2e9856"} Oct 30 18:03:42.690536 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.690516 4455 scope.go:122] "RemoveContainer" containerID="f0e3a178a01ba2f82e89a148e2351926b9d8b177067d306d43a0c391f1b52cee" Oct 30 18:03:42.690761 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.690742 4455 generic.go:356] "Generic (PLEG): container finished" podID="ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b" containerID="c97e0e31dd7438816d7ca218adceb78c8ae3df260360f707dbde7f6c39fc0bb5" exitCode=0 Oct 30 18:03:42.690915 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:03:42.690796 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0e3a178a01ba2f82e89a148e2351926b9d8b177067d306d43a0c391f1b52cee\": container with ID starting with f0e3a178a01ba2f82e89a148e2351926b9d8b177067d306d43a0c391f1b52cee not found: ID does not exist" containerID="f0e3a178a01ba2f82e89a148e2351926b9d8b177067d306d43a0c391f1b52cee" Oct 30 18:03:42.691010 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.690928 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0e3a178a01ba2f82e89a148e2351926b9d8b177067d306d43a0c391f1b52cee"} err="failed to get container status \"f0e3a178a01ba2f82e89a148e2351926b9d8b177067d306d43a0c391f1b52cee\": rpc error: code = NotFound desc = could not find container \"f0e3a178a01ba2f82e89a148e2351926b9d8b177067d306d43a0c391f1b52cee\": container with ID starting with f0e3a178a01ba2f82e89a148e2351926b9d8b177067d306d43a0c391f1b52cee not found: ID does not exist" Oct 30 18:03:42.691010 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.690861 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ce05c7ef-857cf887b6-8knn4" event={"ID":"ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b","Type":"ContainerDied","Data":"c97e0e31dd7438816d7ca218adceb78c8ae3df260360f707dbde7f6c39fc0bb5"} Oct 30 18:03:42.691132 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.691007 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ce05c7ef-857cf887b6-8knn4" event={"ID":"ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b","Type":"ContainerDied","Data":"6e2be04640a0700ae1c2c0863280efcf8c7be602a893e2c8b79d5e163731ad36"} Oct 30 18:03:42.691132 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.690861 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ce05c7ef-857cf887b6-8knn4" Oct 30 18:03:42.691132 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.691028 4455 scope.go:122] "RemoveContainer" containerID="c97e0e31dd7438816d7ca218adceb78c8ae3df260360f707dbde7f6c39fc0bb5" Oct 30 18:03:42.700879 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.698007 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5e0da928-66b7947b4c-8lsn2"] Oct 30 18:03:42.725290 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.725273 4455 scope.go:122] "RemoveContainer" containerID="c97e0e31dd7438816d7ca218adceb78c8ae3df260360f707dbde7f6c39fc0bb5" Oct 30 18:03:42.725639 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:03:42.725620 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c97e0e31dd7438816d7ca218adceb78c8ae3df260360f707dbde7f6c39fc0bb5\": container with ID starting with c97e0e31dd7438816d7ca218adceb78c8ae3df260360f707dbde7f6c39fc0bb5 not found: ID does not exist" containerID="c97e0e31dd7438816d7ca218adceb78c8ae3df260360f707dbde7f6c39fc0bb5" Oct 30 18:03:42.725707 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.725649 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c97e0e31dd7438816d7ca218adceb78c8ae3df260360f707dbde7f6c39fc0bb5"} err="failed to get container status \"c97e0e31dd7438816d7ca218adceb78c8ae3df260360f707dbde7f6c39fc0bb5\": rpc error: code = NotFound desc = could not find container \"c97e0e31dd7438816d7ca218adceb78c8ae3df260360f707dbde7f6c39fc0bb5\": container with ID starting with c97e0e31dd7438816d7ca218adceb78c8ae3df260360f707dbde7f6c39fc0bb5 not found: ID does not exist" Oct 30 18:03:42.733471 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.733446 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce05c7ef-59894df599-q2kd8" Oct 30 18:03:42.751310 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.751280 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d"] Oct 30 18:03:42.769522 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.769487 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a8c61e05-7dccb587d7-s4k4d"] Oct 30 18:03:42.825026 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.824992 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a51e90d-5cff-414d-baca-15fd155cea22-kube-api-access-pmp59\" (UniqueName: \"kubernetes.io/projected/6a51e90d-5cff-414d-baca-15fd155cea22-kube-api-access-pmp59\") pod \"6a51e90d-5cff-414d-baca-15fd155cea22\" (UID: \"6a51e90d-5cff-414d-baca-15fd155cea22\") " Oct 30 18:03:42.825226 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.825078 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b-plugins-config-volume\") pod \"ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b\" (UID: \"ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b\") " Oct 30 18:03:42.825302 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.825266 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b-kube-api-access-d6lpb\" (UniqueName: \"kubernetes.io/projected/ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b-kube-api-access-d6lpb\") pod \"ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b\" (UID: \"ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b\") " Oct 30 18:03:42.825582 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.825542 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b-plugins-config-volume" pod "ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b" (UID: "ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:03:42.825716 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.825695 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a51e90d-5cff-414d-baca-15fd155cea22-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a51e90d-5cff-414d-baca-15fd155cea22-plugins-config-volume\") pod \"6a51e90d-5cff-414d-baca-15fd155cea22\" (UID: \"6a51e90d-5cff-414d-baca-15fd155cea22\") " Oct 30 18:03:42.826082 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.826059 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a51e90d-5cff-414d-baca-15fd155cea22-plugins-config-volume" pod "6a51e90d-5cff-414d-baca-15fd155cea22" (UID: "6a51e90d-5cff-414d-baca-15fd155cea22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:03:42.826586 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.826559 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a51e90d-5cff-414d-baca-15fd155cea22-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:03:42.826668 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.826588 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:03:42.828476 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.828454 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a51e90d-5cff-414d-baca-15fd155cea22-kube-api-access-pmp59" pod "6a51e90d-5cff-414d-baca-15fd155cea22" (UID: "6a51e90d-5cff-414d-baca-15fd155cea22"). InnerVolumeSpecName "kube-api-access-pmp59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:42.828969 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.828950 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b-kube-api-access-d6lpb" pod "ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b" (UID: "ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b"). InnerVolumeSpecName "kube-api-access-d6lpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:42.927832 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.927754 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6lpb\" (UniqueName: \"kubernetes.io/projected/ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b-kube-api-access-d6lpb\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:03:42.927832 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:42.927781 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmp59\" (UniqueName: \"kubernetes.io/projected/6a51e90d-5cff-414d-baca-15fd155cea22-kube-api-access-pmp59\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:03:43.049144 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:43.049108 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ce05c7ef-857cf887b6-8knn4"] Oct 30 18:03:43.055431 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:43.055403 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ce05c7ef-857cf887b6-8knn4"] Oct 30 18:03:43.638561 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:43.638517 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28d593a5-d995-4a16-95d1-e2b349333657" path="/var/lib/kubelet/pods/28d593a5-d995-4a16-95d1-e2b349333657/volumes" Oct 30 18:03:43.639187 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:43.639165 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="343886fc-aa2f-47aa-93c6-b9b7f4627ce4" path="/var/lib/kubelet/pods/343886fc-aa2f-47aa-93c6-b9b7f4627ce4/volumes" Oct 30 18:03:43.639802 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:43.639757 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="648a2b61-90ed-4953-9974-d636fc0db13e" path="/var/lib/kubelet/pods/648a2b61-90ed-4953-9974-d636fc0db13e/volumes" Oct 30 18:03:43.645358 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:43.645312 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abdab22c-d4d9-4673-af5e-fc3388d68fa2" path="/var/lib/kubelet/pods/abdab22c-d4d9-4673-af5e-fc3388d68fa2/volumes" Oct 30 18:03:43.646203 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:43.646173 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca6dae74-7199-4366-b7cc-c2ca15f2001c" path="/var/lib/kubelet/pods/ca6dae74-7199-4366-b7cc-c2ca15f2001c/volumes" Oct 30 18:03:43.646606 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:43.646591 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b" path="/var/lib/kubelet/pods/ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b/volumes" Oct 30 18:03:43.698889 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:43.698787 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce05c7ef-59894df599-q2kd8" Oct 30 18:03:43.698889 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:43.698791 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce05c7ef-59894df599-q2kd8" event={"ID":"6a51e90d-5cff-414d-baca-15fd155cea22","Type":"ContainerDied","Data":"ec1c08b8b7b853c4b5fc20316c4d9e8c3b0434ecc4cc1770d79e538b59a7b1ae"} Oct 30 18:03:43.698889 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:43.698861 4455 scope.go:122] "RemoveContainer" containerID="2b9bbabd16468f1bab4dbb674da21a50c1a935d70422faf6a52c9ba18b2e9856" Oct 30 18:03:43.752476 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:43.752441 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ce05c7ef-59894df599-q2kd8"] Oct 30 18:03:43.755425 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:43.755398 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ce05c7ef-59894df599-q2kd8"] Oct 30 18:03:45.536471 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:45.536423 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a8c61e05-7f949cc5dd-9n6lk" podUID="52e56ab4-42e3-4f11-87f0-1386b3c15ca2" containerName="payload-processing" containerID="cri-o://73e1d1015056eeb0c4b6f1d90eb9b8fb8bdab9e303c976f9878abb09bb01c860" gracePeriod=25 Oct 30 18:03:45.580364 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:45.578418 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a8c61e05-58c947cf67-f4f6f" podUID="a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f" containerName="payload-pre-processing" containerID="cri-o://ebef9631556dfadb115ff8248106b077f2178292c52151924ff1a118c6b24a02" gracePeriod=25 Oct 30 18:03:45.636042 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:45.636009 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a51e90d-5cff-414d-baca-15fd155cea22" path="/var/lib/kubelet/pods/6a51e90d-5cff-414d-baca-15fd155cea22/volumes" Oct 30 18:03:45.720929 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:45.720698 4455 generic.go:356] "Generic (PLEG): container finished" podID="a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f" containerID="ebef9631556dfadb115ff8248106b077f2178292c52151924ff1a118c6b24a02" exitCode=0 Oct 30 18:03:45.720929 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:45.720832 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a8c61e05-58c947cf67-f4f6f" event={"ID":"a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f","Type":"ContainerDied","Data":"ebef9631556dfadb115ff8248106b077f2178292c52151924ff1a118c6b24a02"} Oct 30 18:03:45.723704 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:45.723504 4455 generic.go:356] "Generic (PLEG): container finished" podID="52e56ab4-42e3-4f11-87f0-1386b3c15ca2" containerID="73e1d1015056eeb0c4b6f1d90eb9b8fb8bdab9e303c976f9878abb09bb01c860" exitCode=0 Oct 30 18:03:45.723704 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:45.723639 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a8c61e05-7f949cc5dd-9n6lk" event={"ID":"52e56ab4-42e3-4f11-87f0-1386b3c15ca2","Type":"ContainerDied","Data":"73e1d1015056eeb0c4b6f1d90eb9b8fb8bdab9e303c976f9878abb09bb01c860"} Oct 30 18:03:45.926077 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:45.926052 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a8c61e05-7f949cc5dd-9n6lk" Oct 30 18:03:45.988018 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:45.987985 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a8c61e05-58c947cf67-f4f6f" Oct 30 18:03:46.059499 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.059463 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c"] Oct 30 18:03:46.059726 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.059698 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c" podUID="a887a798-03ec-426a-894a-aeef866f7f26" containerName="maas-api" containerID="cri-o://ca5460d010bd79a09e4d3c971f871834433e6ff7592f2452321e50d89f7119b7" gracePeriod=30 Oct 30 18:03:46.082207 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.082181 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52e56ab4-42e3-4f11-87f0-1386b3c15ca2-kube-api-access-2m9dm\" (UniqueName: \"kubernetes.io/projected/52e56ab4-42e3-4f11-87f0-1386b3c15ca2-kube-api-access-2m9dm\") pod \"52e56ab4-42e3-4f11-87f0-1386b3c15ca2\" (UID: \"52e56ab4-42e3-4f11-87f0-1386b3c15ca2\") " Oct 30 18:03:46.082378 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.082224 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f-plugins-config-volume\") pod \"a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f\" (UID: \"a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f\") " Oct 30 18:03:46.082378 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.082274 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f-kube-api-access-m87cj\" (UniqueName: \"kubernetes.io/projected/a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f-kube-api-access-m87cj\") pod \"a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f\" (UID: \"a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f\") " Oct 30 18:03:46.082508 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.082387 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52e56ab4-42e3-4f11-87f0-1386b3c15ca2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52e56ab4-42e3-4f11-87f0-1386b3c15ca2-plugins-config-volume\") pod \"52e56ab4-42e3-4f11-87f0-1386b3c15ca2\" (UID: \"52e56ab4-42e3-4f11-87f0-1386b3c15ca2\") " Oct 30 18:03:46.082716 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.082681 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f-plugins-config-volume" pod "a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f" (UID: "a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:03:46.082947 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.082910 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52e56ab4-42e3-4f11-87f0-1386b3c15ca2-plugins-config-volume" pod "52e56ab4-42e3-4f11-87f0-1386b3c15ca2" (UID: "52e56ab4-42e3-4f11-87f0-1386b3c15ca2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:03:46.082947 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.082925 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:03:46.084746 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.084723 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f-kube-api-access-m87cj" pod "a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f" (UID: "a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f"). InnerVolumeSpecName "kube-api-access-m87cj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:46.084874 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.084854 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52e56ab4-42e3-4f11-87f0-1386b3c15ca2-kube-api-access-2m9dm" pod "52e56ab4-42e3-4f11-87f0-1386b3c15ca2" (UID: "52e56ab4-42e3-4f11-87f0-1386b3c15ca2"). InnerVolumeSpecName "kube-api-access-2m9dm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:46.183671 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.183643 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m9dm\" (UniqueName: \"kubernetes.io/projected/52e56ab4-42e3-4f11-87f0-1386b3c15ca2-kube-api-access-2m9dm\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:03:46.183826 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.183671 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m87cj\" (UniqueName: \"kubernetes.io/projected/a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f-kube-api-access-m87cj\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:03:46.183826 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.183729 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52e56ab4-42e3-4f11-87f0-1386b3c15ca2-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:03:46.310880 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.310855 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c" Oct 30 18:03:46.384897 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.384863 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a887a798-03ec-426a-894a-aeef866f7f26-kube-api-access-qlj8s\" (UniqueName: \"kubernetes.io/projected/a887a798-03ec-426a-894a-aeef866f7f26-kube-api-access-qlj8s\") pod \"a887a798-03ec-426a-894a-aeef866f7f26\" (UID: \"a887a798-03ec-426a-894a-aeef866f7f26\") " Oct 30 18:03:46.385053 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.384961 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a887a798-03ec-426a-894a-aeef866f7f26-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a887a798-03ec-426a-894a-aeef866f7f26-maas-api-tls\") pod \"a887a798-03ec-426a-894a-aeef866f7f26\" (UID: \"a887a798-03ec-426a-894a-aeef866f7f26\") " Oct 30 18:03:46.385053 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.385021 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a887a798-03ec-426a-894a-aeef866f7f26-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a887a798-03ec-426a-894a-aeef866f7f26-metrics-certs\") pod \"a887a798-03ec-426a-894a-aeef866f7f26\" (UID: \"a887a798-03ec-426a-894a-aeef866f7f26\") " Oct 30 18:03:46.385129 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.385058 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a887a798-03ec-426a-894a-aeef866f7f26-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a887a798-03ec-426a-894a-aeef866f7f26-openshift-service-ca\") pod \"a887a798-03ec-426a-894a-aeef866f7f26\" (UID: \"a887a798-03ec-426a-894a-aeef866f7f26\") " Oct 30 18:03:46.385640 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.385547 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a887a798-03ec-426a-894a-aeef866f7f26-openshift-service-ca" pod "a887a798-03ec-426a-894a-aeef866f7f26" (UID: "a887a798-03ec-426a-894a-aeef866f7f26"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:03:46.387250 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.387218 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a887a798-03ec-426a-894a-aeef866f7f26-kube-api-access-qlj8s" pod "a887a798-03ec-426a-894a-aeef866f7f26" (UID: "a887a798-03ec-426a-894a-aeef866f7f26"). InnerVolumeSpecName "kube-api-access-qlj8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:46.387250 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.387225 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a887a798-03ec-426a-894a-aeef866f7f26-metrics-certs" pod "a887a798-03ec-426a-894a-aeef866f7f26" (UID: "a887a798-03ec-426a-894a-aeef866f7f26"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:03:46.387250 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.387241 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a887a798-03ec-426a-894a-aeef866f7f26-maas-api-tls" pod "a887a798-03ec-426a-894a-aeef866f7f26" (UID: "a887a798-03ec-426a-894a-aeef866f7f26"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:03:46.486652 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.486617 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a887a798-03ec-426a-894a-aeef866f7f26-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:03:46.486652 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.486644 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a887a798-03ec-426a-894a-aeef866f7f26-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:03:46.486652 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.486653 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a887a798-03ec-426a-894a-aeef866f7f26-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:03:46.486878 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.486662 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlj8s\" (UniqueName: \"kubernetes.io/projected/a887a798-03ec-426a-894a-aeef866f7f26-kube-api-access-qlj8s\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:03:46.734435 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.734401 4455 generic.go:356] "Generic (PLEG): container finished" podID="a887a798-03ec-426a-894a-aeef866f7f26" containerID="ca5460d010bd79a09e4d3c971f871834433e6ff7592f2452321e50d89f7119b7" exitCode=0 Oct 30 18:03:46.734983 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.734452 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c" Oct 30 18:03:46.734983 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.734486 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c" event={"ID":"a887a798-03ec-426a-894a-aeef866f7f26","Type":"ContainerDied","Data":"ca5460d010bd79a09e4d3c971f871834433e6ff7592f2452321e50d89f7119b7"} Oct 30 18:03:46.734983 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.734580 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c" event={"ID":"a887a798-03ec-426a-894a-aeef866f7f26","Type":"ContainerDied","Data":"ea42af0b44f4aa9386dac402b31673d17fca5955b31d9b7f0cd623fc2cc12bc0"} Oct 30 18:03:46.734983 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.734612 4455 scope.go:122] "RemoveContainer" containerID="ca5460d010bd79a09e4d3c971f871834433e6ff7592f2452321e50d89f7119b7" Oct 30 18:03:46.736775 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.736699 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a8c61e05-58c947cf67-f4f6f" event={"ID":"a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f","Type":"ContainerDied","Data":"299789d32a179a20b2743fd34c9f4dbf197a189c6936baa62ff9c1ed40080cd9"} Oct 30 18:03:46.736906 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.736794 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a8c61e05-58c947cf67-f4f6f" Oct 30 18:03:46.740889 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.740854 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a8c61e05-7f949cc5dd-9n6lk" event={"ID":"52e56ab4-42e3-4f11-87f0-1386b3c15ca2","Type":"ContainerDied","Data":"60cc8def0c9744c3b6e569d06be3030e72968d51f50a83d1e95da3a0e8a7106a"} Oct 30 18:03:46.740985 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.740961 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a8c61e05-7f949cc5dd-9n6lk" Oct 30 18:03:46.748444 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.748384 4455 scope.go:122] "RemoveContainer" containerID="ca5460d010bd79a09e4d3c971f871834433e6ff7592f2452321e50d89f7119b7" Oct 30 18:03:46.748730 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:03:46.748701 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca5460d010bd79a09e4d3c971f871834433e6ff7592f2452321e50d89f7119b7\": container with ID starting with ca5460d010bd79a09e4d3c971f871834433e6ff7592f2452321e50d89f7119b7 not found: ID does not exist" containerID="ca5460d010bd79a09e4d3c971f871834433e6ff7592f2452321e50d89f7119b7" Oct 30 18:03:46.748838 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.748741 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca5460d010bd79a09e4d3c971f871834433e6ff7592f2452321e50d89f7119b7"} err="failed to get container status \"ca5460d010bd79a09e4d3c971f871834433e6ff7592f2452321e50d89f7119b7\": rpc error: code = NotFound desc = could not find container \"ca5460d010bd79a09e4d3c971f871834433e6ff7592f2452321e50d89f7119b7\": container with ID starting with ca5460d010bd79a09e4d3c971f871834433e6ff7592f2452321e50d89f7119b7 not found: ID does not exist" Oct 30 18:03:46.748838 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.748766 4455 scope.go:122] "RemoveContainer" containerID="ebef9631556dfadb115ff8248106b077f2178292c52151924ff1a118c6b24a02" Oct 30 18:03:46.758259 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.758236 4455 scope.go:122] "RemoveContainer" containerID="73e1d1015056eeb0c4b6f1d90eb9b8fb8bdab9e303c976f9878abb09bb01c860" Oct 30 18:03:46.806513 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.806482 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c"] Oct 30 18:03:46.817236 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.817217 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5e0da928-86cd4cffff-ngs6c"] Oct 30 18:03:46.862897 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.862858 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a8c61e05-58c947cf67-f4f6f"] Oct 30 18:03:46.865838 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.865813 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a8c61e05-58c947cf67-f4f6f"] Oct 30 18:03:46.902529 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.902497 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a8c61e05-7f949cc5dd-9n6lk"] Oct 30 18:03:46.907218 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:46.907186 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a8c61e05-7f949cc5dd-9n6lk"] Oct 30 18:03:47.633655 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:47.633618 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52e56ab4-42e3-4f11-87f0-1386b3c15ca2" path="/var/lib/kubelet/pods/52e56ab4-42e3-4f11-87f0-1386b3c15ca2/volumes" Oct 30 18:03:47.634112 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:47.634091 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f" path="/var/lib/kubelet/pods/a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f/volumes" Oct 30 18:03:47.634602 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:47.634580 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a887a798-03ec-426a-894a-aeef866f7f26" path="/var/lib/kubelet/pods/a887a798-03ec-426a-894a-aeef866f7f26/volumes" Oct 30 18:03:47.663538 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:47.663504 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-5e0da928-7d8b7dddcb-5q6z8" podUID="4e4645b3-3e3d-4330-946b-39543b416f48" containerName="payload-processing" containerID="cri-o://93e6e73b8543fbd54ff7728aa54179db5780cb3acd61119d7d5b493a77c2c68f" gracePeriod=25 Oct 30 18:03:47.699201 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:47.699169 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-5e0da928-66b7947b4c-8lsn2" podUID="c253589b-8899-46ee-b9db-9ae7e05d0571" containerName="payload-pre-processing" containerID="cri-o://80b0a03d5036ca521ebf8eee555e5f3c7cc260ff30fb8c43bfec67daec326a28" gracePeriod=25 Oct 30 18:03:47.969557 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:47.969525 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5e0da928-7d8b7dddcb-5q6z8" Oct 30 18:03:48.002966 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:48.002944 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4645b3-3e3d-4330-946b-39543b416f48-kube-api-access-9tp44\" (UniqueName: \"kubernetes.io/projected/4e4645b3-3e3d-4330-946b-39543b416f48-kube-api-access-9tp44\") pod \"4e4645b3-3e3d-4330-946b-39543b416f48\" (UID: \"4e4645b3-3e3d-4330-946b-39543b416f48\") " Oct 30 18:03:48.003118 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:48.003091 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e4645b3-3e3d-4330-946b-39543b416f48-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4645b3-3e3d-4330-946b-39543b416f48-plugins-config-volume\") pod \"4e4645b3-3e3d-4330-946b-39543b416f48\" (UID: \"4e4645b3-3e3d-4330-946b-39543b416f48\") " Oct 30 18:03:48.003441 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:48.003421 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e4645b3-3e3d-4330-946b-39543b416f48-plugins-config-volume" pod "4e4645b3-3e3d-4330-946b-39543b416f48" (UID: "4e4645b3-3e3d-4330-946b-39543b416f48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:03:48.005301 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:48.005274 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4645b3-3e3d-4330-946b-39543b416f48-kube-api-access-9tp44" pod "4e4645b3-3e3d-4330-946b-39543b416f48" (UID: "4e4645b3-3e3d-4330-946b-39543b416f48"). InnerVolumeSpecName "kube-api-access-9tp44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:48.008576 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:48.008556 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5e0da928-66b7947b4c-8lsn2" Oct 30 18:03:48.104097 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:48.104063 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c253589b-8899-46ee-b9db-9ae7e05d0571-kube-api-access-tx5cf\" (UniqueName: \"kubernetes.io/projected/c253589b-8899-46ee-b9db-9ae7e05d0571-kube-api-access-tx5cf\") pod \"c253589b-8899-46ee-b9db-9ae7e05d0571\" (UID: \"c253589b-8899-46ee-b9db-9ae7e05d0571\") " Oct 30 18:03:48.104284 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:48.104124 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c253589b-8899-46ee-b9db-9ae7e05d0571-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c253589b-8899-46ee-b9db-9ae7e05d0571-plugins-config-volume\") pod \"c253589b-8899-46ee-b9db-9ae7e05d0571\" (UID: \"c253589b-8899-46ee-b9db-9ae7e05d0571\") " Oct 30 18:03:48.104486 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:48.104470 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4645b3-3e3d-4330-946b-39543b416f48-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:03:48.104564 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:48.104490 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tp44\" (UniqueName: \"kubernetes.io/projected/4e4645b3-3e3d-4330-946b-39543b416f48-kube-api-access-9tp44\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:03:48.104564 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:48.104507 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c253589b-8899-46ee-b9db-9ae7e05d0571-plugins-config-volume" pod "c253589b-8899-46ee-b9db-9ae7e05d0571" (UID: "c253589b-8899-46ee-b9db-9ae7e05d0571"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:03:48.106477 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:48.106448 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c253589b-8899-46ee-b9db-9ae7e05d0571-kube-api-access-tx5cf" pod "c253589b-8899-46ee-b9db-9ae7e05d0571" (UID: "c253589b-8899-46ee-b9db-9ae7e05d0571"). InnerVolumeSpecName "kube-api-access-tx5cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:48.205491 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:48.205415 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c253589b-8899-46ee-b9db-9ae7e05d0571-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:03:48.205491 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:48.205442 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx5cf\" (UniqueName: \"kubernetes.io/projected/c253589b-8899-46ee-b9db-9ae7e05d0571-kube-api-access-tx5cf\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:03:48.771688 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:48.771657 4455 generic.go:356] "Generic (PLEG): container finished" podID="c253589b-8899-46ee-b9db-9ae7e05d0571" containerID="80b0a03d5036ca521ebf8eee555e5f3c7cc260ff30fb8c43bfec67daec326a28" exitCode=0 Oct 30 18:03:48.771877 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:48.771706 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5e0da928-66b7947b4c-8lsn2" Oct 30 18:03:48.771877 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:48.771744 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5e0da928-66b7947b4c-8lsn2" event={"ID":"c253589b-8899-46ee-b9db-9ae7e05d0571","Type":"ContainerDied","Data":"80b0a03d5036ca521ebf8eee555e5f3c7cc260ff30fb8c43bfec67daec326a28"} Oct 30 18:03:48.771877 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:48.771807 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5e0da928-66b7947b4c-8lsn2" event={"ID":"c253589b-8899-46ee-b9db-9ae7e05d0571","Type":"ContainerDied","Data":"5865916019d488844dca0a3f6da96f06f1a08766b8a638c556517e38664f25d7"} Oct 30 18:03:48.771877 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:48.771835 4455 scope.go:122] "RemoveContainer" containerID="80b0a03d5036ca521ebf8eee555e5f3c7cc260ff30fb8c43bfec67daec326a28" Oct 30 18:03:48.773499 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:48.773473 4455 generic.go:356] "Generic (PLEG): container finished" podID="4e4645b3-3e3d-4330-946b-39543b416f48" containerID="93e6e73b8543fbd54ff7728aa54179db5780cb3acd61119d7d5b493a77c2c68f" exitCode=0 Oct 30 18:03:48.773622 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:48.773522 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5e0da928-7d8b7dddcb-5q6z8" Oct 30 18:03:48.773622 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:48.773579 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5e0da928-7d8b7dddcb-5q6z8" event={"ID":"4e4645b3-3e3d-4330-946b-39543b416f48","Type":"ContainerDied","Data":"93e6e73b8543fbd54ff7728aa54179db5780cb3acd61119d7d5b493a77c2c68f"} Oct 30 18:03:48.773729 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:48.773620 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5e0da928-7d8b7dddcb-5q6z8" event={"ID":"4e4645b3-3e3d-4330-946b-39543b416f48","Type":"ContainerDied","Data":"22c67080261314f1fb28abc407add2e09208deed2157bdf2d76117034f69272f"} Oct 30 18:03:48.782814 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:48.782800 4455 scope.go:122] "RemoveContainer" containerID="80b0a03d5036ca521ebf8eee555e5f3c7cc260ff30fb8c43bfec67daec326a28" Oct 30 18:03:48.783185 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:03:48.783116 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80b0a03d5036ca521ebf8eee555e5f3c7cc260ff30fb8c43bfec67daec326a28\": container with ID starting with 80b0a03d5036ca521ebf8eee555e5f3c7cc260ff30fb8c43bfec67daec326a28 not found: ID does not exist" containerID="80b0a03d5036ca521ebf8eee555e5f3c7cc260ff30fb8c43bfec67daec326a28" Oct 30 18:03:48.783185 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:48.783151 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80b0a03d5036ca521ebf8eee555e5f3c7cc260ff30fb8c43bfec67daec326a28"} err="failed to get container status \"80b0a03d5036ca521ebf8eee555e5f3c7cc260ff30fb8c43bfec67daec326a28\": rpc error: code = NotFound desc = could not find container \"80b0a03d5036ca521ebf8eee555e5f3c7cc260ff30fb8c43bfec67daec326a28\": container with ID starting with 80b0a03d5036ca521ebf8eee555e5f3c7cc260ff30fb8c43bfec67daec326a28 not found: ID does not exist" Oct 30 18:03:48.783185 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:48.783176 4455 scope.go:122] "RemoveContainer" containerID="93e6e73b8543fbd54ff7728aa54179db5780cb3acd61119d7d5b493a77c2c68f" Oct 30 18:03:48.793895 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:48.793876 4455 scope.go:122] "RemoveContainer" containerID="93e6e73b8543fbd54ff7728aa54179db5780cb3acd61119d7d5b493a77c2c68f" Oct 30 18:03:48.794189 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:03:48.794168 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93e6e73b8543fbd54ff7728aa54179db5780cb3acd61119d7d5b493a77c2c68f\": container with ID starting with 93e6e73b8543fbd54ff7728aa54179db5780cb3acd61119d7d5b493a77c2c68f not found: ID does not exist" containerID="93e6e73b8543fbd54ff7728aa54179db5780cb3acd61119d7d5b493a77c2c68f" Oct 30 18:03:48.794266 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:48.794199 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93e6e73b8543fbd54ff7728aa54179db5780cb3acd61119d7d5b493a77c2c68f"} err="failed to get container status \"93e6e73b8543fbd54ff7728aa54179db5780cb3acd61119d7d5b493a77c2c68f\": rpc error: code = NotFound desc = could not find container \"93e6e73b8543fbd54ff7728aa54179db5780cb3acd61119d7d5b493a77c2c68f\": container with ID starting with 93e6e73b8543fbd54ff7728aa54179db5780cb3acd61119d7d5b493a77c2c68f not found: ID does not exist" Oct 30 18:03:48.837153 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:48.837115 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5e0da928-7d8b7dddcb-5q6z8"] Oct 30 18:03:48.847001 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:48.846972 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5e0da928-7d8b7dddcb-5q6z8"] Oct 30 18:03:48.877032 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:48.876996 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5e0da928-66b7947b4c-8lsn2"] Oct 30 18:03:48.884850 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:48.884816 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5e0da928-66b7947b4c-8lsn2"] Oct 30 18:03:49.465682 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.465651 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-6e8fca-79548bc54c-5nw4l"] Oct 30 18:03:49.466148 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466134 4455 cpu_manager.go:414] "containerMap: removing container" podUID="a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f" containerName="payload-pre-processing" Oct 30 18:03:49.466202 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466150 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f" containerName="payload-pre-processing" Oct 30 18:03:49.466202 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466163 4455 cpu_manager.go:414] "containerMap: removing container" podUID="c253589b-8899-46ee-b9db-9ae7e05d0571" containerName="payload-pre-processing" Oct 30 18:03:49.466202 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466169 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c253589b-8899-46ee-b9db-9ae7e05d0571" containerName="payload-pre-processing" Oct 30 18:03:49.466202 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466177 4455 cpu_manager.go:414] "containerMap: removing container" podUID="ca6dae74-7199-4366-b7cc-c2ca15f2001c" containerName="payload-pre-processing" Oct 30 18:03:49.466202 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466182 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca6dae74-7199-4366-b7cc-c2ca15f2001c" containerName="payload-pre-processing" Oct 30 18:03:49.466202 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466190 4455 cpu_manager.go:414] "containerMap: removing container" podUID="343886fc-aa2f-47aa-93c6-b9b7f4627ce4" containerName="payload-processing" Oct 30 18:03:49.466202 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466195 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="343886fc-aa2f-47aa-93c6-b9b7f4627ce4" containerName="payload-processing" Oct 30 18:03:49.466202 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466202 4455 cpu_manager.go:414] "containerMap: removing container" podUID="28d593a5-d995-4a16-95d1-e2b349333657" containerName="maas-api" Oct 30 18:03:49.466472 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466206 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28d593a5-d995-4a16-95d1-e2b349333657" containerName="maas-api" Oct 30 18:03:49.466472 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466213 4455 cpu_manager.go:414] "containerMap: removing container" podUID="4e4645b3-3e3d-4330-946b-39543b416f48" containerName="payload-processing" Oct 30 18:03:49.466472 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466217 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e4645b3-3e3d-4330-946b-39543b416f48" containerName="payload-processing" Oct 30 18:03:49.466472 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466226 4455 cpu_manager.go:414] "containerMap: removing container" podUID="abdab22c-d4d9-4673-af5e-fc3388d68fa2" containerName="maas-api" Oct 30 18:03:49.466472 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466231 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abdab22c-d4d9-4673-af5e-fc3388d68fa2" containerName="maas-api" Oct 30 18:03:49.466472 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466240 4455 cpu_manager.go:414] "containerMap: removing container" podUID="52e56ab4-42e3-4f11-87f0-1386b3c15ca2" containerName="payload-processing" Oct 30 18:03:49.466472 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466244 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52e56ab4-42e3-4f11-87f0-1386b3c15ca2" containerName="payload-processing" Oct 30 18:03:49.466472 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466254 4455 cpu_manager.go:414] "containerMap: removing container" podUID="648a2b61-90ed-4953-9974-d636fc0db13e" containerName="maas-api" Oct 30 18:03:49.466472 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466259 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="648a2b61-90ed-4953-9974-d636fc0db13e" containerName="maas-api" Oct 30 18:03:49.466472 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466275 4455 cpu_manager.go:414] "containerMap: removing container" podUID="ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b" containerName="payload-processing" Oct 30 18:03:49.466472 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466280 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b" containerName="payload-processing" Oct 30 18:03:49.466472 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466287 4455 cpu_manager.go:414] "containerMap: removing container" podUID="a887a798-03ec-426a-894a-aeef866f7f26" containerName="maas-api" Oct 30 18:03:49.466472 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466291 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a887a798-03ec-426a-894a-aeef866f7f26" containerName="maas-api" Oct 30 18:03:49.466472 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466303 4455 cpu_manager.go:414] "containerMap: removing container" podUID="6a51e90d-5cff-414d-baca-15fd155cea22" containerName="payload-pre-processing" Oct 30 18:03:49.466472 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466307 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a51e90d-5cff-414d-baca-15fd155cea22" containerName="payload-pre-processing" Oct 30 18:03:49.466472 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466414 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca6dae74-7199-4366-b7cc-c2ca15f2001c" containerName="payload-pre-processing" Oct 30 18:03:49.466472 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466423 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="343886fc-aa2f-47aa-93c6-b9b7f4627ce4" containerName="payload-processing" Oct 30 18:03:49.466472 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466430 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="28d593a5-d995-4a16-95d1-e2b349333657" containerName="maas-api" Oct 30 18:03:49.466472 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466435 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e4645b3-3e3d-4330-946b-39543b416f48" containerName="payload-processing" Oct 30 18:03:49.466472 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466440 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="abdab22c-d4d9-4673-af5e-fc3388d68fa2" containerName="maas-api" Oct 30 18:03:49.466472 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466446 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="52e56ab4-42e3-4f11-87f0-1386b3c15ca2" containerName="payload-processing" Oct 30 18:03:49.466472 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466452 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="648a2b61-90ed-4953-9974-d636fc0db13e" containerName="maas-api" Oct 30 18:03:49.466472 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466457 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="a887a798-03ec-426a-894a-aeef866f7f26" containerName="maas-api" Oct 30 18:03:49.466472 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466464 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec808be2-6a7b-4ac6-bdd4-7dc9b62bb50b" containerName="payload-processing" Oct 30 18:03:49.466472 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466470 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6d0ea3f-0b58-48e1-ab6c-6b45871fbd9f" containerName="payload-pre-processing" Oct 30 18:03:49.466472 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466476 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a51e90d-5cff-414d-baca-15fd155cea22" containerName="payload-pre-processing" Oct 30 18:03:49.466472 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.466483 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="c253589b-8899-46ee-b9db-9ae7e05d0571" containerName="payload-pre-processing" Oct 30 18:03:49.471336 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.471302 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6e8fca-79548bc54c-5nw4l" Oct 30 18:03:49.496525 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.496501 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-6e8fca\"" Oct 30 18:03:49.496796 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.496779 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-6e8fca-dockercfg-s5dv6\"" Oct 30 18:03:49.500432 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.500409 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-6e8fca-79548bc54c-5nw4l"] Oct 30 18:03:49.523652 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.523615 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-864nd\" (UniqueName: \"kubernetes.io/projected/b32c8229-35fa-4eff-abf2-4d3b8713b1eb-kube-api-access-864nd\") pod \"payload-pre-processing-e2e-worker-w2-6e8fca-79548bc54c-5nw4l\" (UID: \"b32c8229-35fa-4eff-abf2-4d3b8713b1eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6e8fca-79548bc54c-5nw4l" Oct 30 18:03:49.523824 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.523698 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b32c8229-35fa-4eff-abf2-4d3b8713b1eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-6e8fca-79548bc54c-5nw4l\" (UID: \"b32c8229-35fa-4eff-abf2-4d3b8713b1eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6e8fca-79548bc54c-5nw4l" Oct 30 18:03:49.548476 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.548441 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-6e8fca-7df8c6d488-xwlc2"] Oct 30 18:03:49.553047 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.553018 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-6e8fca-7df8c6d488-xwlc2" Oct 30 18:03:49.586350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.584432 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-6e8fca-7df8c6d488-xwlc2"] Oct 30 18:03:49.624821 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.624780 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8gcs\" (UniqueName: \"kubernetes.io/projected/1b032444-b7c0-4d3c-8a07-aee2df84d8bd-kube-api-access-w8gcs\") pod \"payload-processing-e2e-worker-w2-6e8fca-7df8c6d488-xwlc2\" (UID: \"1b032444-b7c0-4d3c-8a07-aee2df84d8bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-6e8fca-7df8c6d488-xwlc2" Oct 30 18:03:49.624992 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.624890 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-864nd\" (UniqueName: \"kubernetes.io/projected/b32c8229-35fa-4eff-abf2-4d3b8713b1eb-kube-api-access-864nd\") pod \"payload-pre-processing-e2e-worker-w2-6e8fca-79548bc54c-5nw4l\" (UID: \"b32c8229-35fa-4eff-abf2-4d3b8713b1eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6e8fca-79548bc54c-5nw4l" Oct 30 18:03:49.625071 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.624997 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b32c8229-35fa-4eff-abf2-4d3b8713b1eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-6e8fca-79548bc54c-5nw4l\" (UID: \"b32c8229-35fa-4eff-abf2-4d3b8713b1eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6e8fca-79548bc54c-5nw4l" Oct 30 18:03:49.625071 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.625054 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b032444-b7c0-4d3c-8a07-aee2df84d8bd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-6e8fca-7df8c6d488-xwlc2\" (UID: \"1b032444-b7c0-4d3c-8a07-aee2df84d8bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-6e8fca-7df8c6d488-xwlc2" Oct 30 18:03:49.626360 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.626308 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b32c8229-35fa-4eff-abf2-4d3b8713b1eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-6e8fca-79548bc54c-5nw4l\" (UID: \"b32c8229-35fa-4eff-abf2-4d3b8713b1eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6e8fca-79548bc54c-5nw4l" Oct 30 18:03:49.635625 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.635599 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e4645b3-3e3d-4330-946b-39543b416f48" path="/var/lib/kubelet/pods/4e4645b3-3e3d-4330-946b-39543b416f48/volumes" Oct 30 18:03:49.635963 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.635951 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c253589b-8899-46ee-b9db-9ae7e05d0571" path="/var/lib/kubelet/pods/c253589b-8899-46ee-b9db-9ae7e05d0571/volumes" Oct 30 18:03:49.649209 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.649185 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-864nd\" (UniqueName: \"kubernetes.io/projected/b32c8229-35fa-4eff-abf2-4d3b8713b1eb-kube-api-access-864nd\") pod \"payload-pre-processing-e2e-worker-w2-6e8fca-79548bc54c-5nw4l\" (UID: \"b32c8229-35fa-4eff-abf2-4d3b8713b1eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6e8fca-79548bc54c-5nw4l" Oct 30 18:03:49.726603 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.726500 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b032444-b7c0-4d3c-8a07-aee2df84d8bd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-6e8fca-7df8c6d488-xwlc2\" (UID: \"1b032444-b7c0-4d3c-8a07-aee2df84d8bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-6e8fca-7df8c6d488-xwlc2" Oct 30 18:03:49.726780 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.726599 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8gcs\" (UniqueName: \"kubernetes.io/projected/1b032444-b7c0-4d3c-8a07-aee2df84d8bd-kube-api-access-w8gcs\") pod \"payload-processing-e2e-worker-w2-6e8fca-7df8c6d488-xwlc2\" (UID: \"1b032444-b7c0-4d3c-8a07-aee2df84d8bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-6e8fca-7df8c6d488-xwlc2" Oct 30 18:03:49.727291 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.727267 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b032444-b7c0-4d3c-8a07-aee2df84d8bd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-6e8fca-7df8c6d488-xwlc2\" (UID: \"1b032444-b7c0-4d3c-8a07-aee2df84d8bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-6e8fca-7df8c6d488-xwlc2" Oct 30 18:03:49.749252 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.749222 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8gcs\" (UniqueName: \"kubernetes.io/projected/1b032444-b7c0-4d3c-8a07-aee2df84d8bd-kube-api-access-w8gcs\") pod \"payload-processing-e2e-worker-w2-6e8fca-7df8c6d488-xwlc2\" (UID: \"1b032444-b7c0-4d3c-8a07-aee2df84d8bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-6e8fca-7df8c6d488-xwlc2" Oct 30 18:03:49.784016 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.783980 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6e8fca-79548bc54c-5nw4l" Oct 30 18:03:49.894486 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:49.894460 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-6e8fca-7df8c6d488-xwlc2" Oct 30 18:03:50.083721 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:50.083691 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-6e8fca-79548bc54c-5nw4l"] Oct 30 18:03:50.085587 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:03:50.085557 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb32c8229_35fa_4eff_abf2_4d3b8713b1eb.slice/crio-7cbd050f3850552529a1d3fa78a2e3fe088d846fc7eecf773d56a3e350f6c7ad WatchSource:0}: Error finding container 7cbd050f3850552529a1d3fa78a2e3fe088d846fc7eecf773d56a3e350f6c7ad: Status 404 returned error can't find the container with id 7cbd050f3850552529a1d3fa78a2e3fe088d846fc7eecf773d56a3e350f6c7ad Oct 30 18:03:50.210387 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:50.209243 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-6e8fca-7df8c6d488-xwlc2"] Oct 30 18:03:50.792179 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:50.792138 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-6e8fca-7df8c6d488-xwlc2" event={"ID":"1b032444-b7c0-4d3c-8a07-aee2df84d8bd","Type":"ContainerStarted","Data":"a1be86cf28df5221c774c69addf2792ded73fdc761656f0d8e96ad88f7acbb6f"} Oct 30 18:03:50.792668 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:50.792206 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-6e8fca-7df8c6d488-xwlc2" event={"ID":"1b032444-b7c0-4d3c-8a07-aee2df84d8bd","Type":"ContainerStarted","Data":"e7240a7fef6527a0868a3fdedd6bfb053caca928f7958fa321f7309e205d964f"} Oct 30 18:03:50.792668 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:50.792356 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-6e8fca-7df8c6d488-xwlc2" Oct 30 18:03:50.794001 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:50.793977 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6e8fca-79548bc54c-5nw4l" event={"ID":"b32c8229-35fa-4eff-abf2-4d3b8713b1eb","Type":"ContainerStarted","Data":"c4ea9cec19b8c89b5f85f2cf502f07c1bbe6b4fa4e51fa1619385bec8f5069b0"} Oct 30 18:03:50.794141 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:50.794020 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6e8fca-79548bc54c-5nw4l" event={"ID":"b32c8229-35fa-4eff-abf2-4d3b8713b1eb","Type":"ContainerStarted","Data":"7cbd050f3850552529a1d3fa78a2e3fe088d846fc7eecf773d56a3e350f6c7ad"} Oct 30 18:03:50.794141 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:50.794065 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6e8fca-79548bc54c-5nw4l" Oct 30 18:03:50.832783 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:50.832729 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-6e8fca-7df8c6d488-xwlc2" podStartSLOduration=1.8327167979999999 podStartE2EDuration="1.832716798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:50.831875752 +0000 UTC m=+683.319512787" watchObservedRunningTime="2026-10-30 18:03:50.832716798 +0000 UTC m=+683.320353834" Oct 30 18:03:50.882523 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:50.882468 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6e8fca-79548bc54c-5nw4l" podStartSLOduration=1.882452668 podStartE2EDuration="1.882452668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:50.879136214 +0000 UTC m=+683.366773249" watchObservedRunningTime="2026-10-30 18:03:50.882452668 +0000 UTC m=+683.370089702" Oct 30 18:03:51.466070 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:51.466022 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr"] Oct 30 18:03:51.471479 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:51.471447 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr" Oct 30 18:03:51.507405 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:51.504485 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-6e8fca-serving-cert\"" Oct 30 18:03:51.507405 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:51.504548 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 18:03:51.507405 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:51.504570 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8pmq2\"" Oct 30 18:03:51.507714 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:51.507695 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr"] Oct 30 18:03:51.546652 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:51.546625 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5-maas-api-tls\") pod \"maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr\" (UID: \"038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr" Oct 30 18:03:51.546812 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:51.546737 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5-metrics-certs\") pod \"maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr\" (UID: \"038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr" Oct 30 18:03:51.546812 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:51.546762 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr\" (UID: \"038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr" Oct 30 18:03:51.546812 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:51.546793 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49qbf\" (UniqueName: \"kubernetes.io/projected/038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5-kube-api-access-49qbf\") pod \"maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr\" (UID: \"038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr" Oct 30 18:03:51.648147 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:51.648105 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5-metrics-certs\") pod \"maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr\" (UID: \"038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr" Oct 30 18:03:51.648346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:51.648151 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr\" (UID: \"038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr" Oct 30 18:03:51.648346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:51.648183 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49qbf\" (UniqueName: \"kubernetes.io/projected/038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5-kube-api-access-49qbf\") pod \"maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr\" (UID: \"038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr" Oct 30 18:03:51.648346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:51.648298 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5-maas-api-tls\") pod \"maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr\" (UID: \"038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr" Oct 30 18:03:51.648996 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:51.648972 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr\" (UID: \"038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr" Oct 30 18:03:51.650706 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:51.650683 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5-metrics-certs\") pod \"maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr\" (UID: \"038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr" Oct 30 18:03:51.650817 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:51.650752 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5-maas-api-tls\") pod \"maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr\" (UID: \"038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr" Oct 30 18:03:51.674793 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:51.674758 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49qbf\" (UniqueName: \"kubernetes.io/projected/038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5-kube-api-access-49qbf\") pod \"maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr\" (UID: \"038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr" Oct 30 18:03:51.791345 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:51.791232 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr" Oct 30 18:03:52.102811 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:52.102777 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr"] Oct 30 18:03:52.104140 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:03:52.104108 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod038dfdf8_c4a1_4ba3_91bf_b7ee685eb5f5.slice/crio-3615b9710c958f63f5795eb60cee6e31476b9d78f155d01640739f07e2a14cd5 WatchSource:0}: Error finding container 3615b9710c958f63f5795eb60cee6e31476b9d78f155d01640739f07e2a14cd5: Status 404 returned error can't find the container with id 3615b9710c958f63f5795eb60cee6e31476b9d78f155d01640739f07e2a14cd5 Oct 30 18:03:52.808767 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:52.808727 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr" event={"ID":"038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5","Type":"ContainerStarted","Data":"7697d7815fff393e63570d60961c041ed77dbf6235f37519ef6b8e30b35da0bd"} Oct 30 18:03:52.808951 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:52.808795 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr" Oct 30 18:03:52.808951 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:52.808818 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr" event={"ID":"038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5","Type":"ContainerStarted","Data":"3615b9710c958f63f5795eb60cee6e31476b9d78f155d01640739f07e2a14cd5"} Oct 30 18:03:52.857194 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:52.857134 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr" podStartSLOduration=1.57413064 podStartE2EDuration="1.857115732s" totalImagesPullingTime="282.985092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:52.854950178 +0000 UTC m=+685.342587213" watchObservedRunningTime="2026-10-30 18:03:52.857115732 +0000 UTC m=+685.344752768" Oct 30 18:03:53.793065 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:53.793028 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-419d32-6dff5c9996-jcbjp"] Oct 30 18:03:53.797444 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:53.797415 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-419d32-6dff5c9996-jcbjp" Oct 30 18:03:53.804199 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:53.804171 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-419d32-6b9bb548f9-bzxk2"] Oct 30 18:03:53.809364 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:53.809343 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-419d32-6b9bb548f9-bzxk2" Oct 30 18:03:53.813958 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:53.813936 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-419d32\"" Oct 30 18:03:53.825293 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:53.825261 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-419d32-6dff5c9996-jcbjp"] Oct 30 18:03:53.846261 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:53.846228 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-419d32-dockercfg-58r4k\"" Oct 30 18:03:53.849757 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:53.849727 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-419d32-6b9bb548f9-bzxk2"] Oct 30 18:03:53.875915 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:53.875880 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnllz\" (UniqueName: \"kubernetes.io/projected/c1b171c5-abf2-4fd2-9b7f-5d103f44f704-kube-api-access-pnllz\") pod \"payload-processing-e2e-worker-w4-419d32-6dff5c9996-jcbjp\" (UID: \"c1b171c5-abf2-4fd2-9b7f-5d103f44f704\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-419d32-6dff5c9996-jcbjp" Oct 30 18:03:53.876095 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:53.875924 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b171c5-abf2-4fd2-9b7f-5d103f44f704-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-419d32-6dff5c9996-jcbjp\" (UID: \"c1b171c5-abf2-4fd2-9b7f-5d103f44f704\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-419d32-6dff5c9996-jcbjp" Oct 30 18:03:53.876164 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:53.876118 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f13fc8d-3df1-4f51-816e-69f10c2a03cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-419d32-6b9bb548f9-bzxk2\" (UID: \"3f13fc8d-3df1-4f51-816e-69f10c2a03cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-419d32-6b9bb548f9-bzxk2" Oct 30 18:03:53.876226 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:53.876172 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw6cp\" (UniqueName: \"kubernetes.io/projected/3f13fc8d-3df1-4f51-816e-69f10c2a03cb-kube-api-access-gw6cp\") pod \"payload-pre-processing-e2e-worker-w4-419d32-6b9bb548f9-bzxk2\" (UID: \"3f13fc8d-3df1-4f51-816e-69f10c2a03cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-419d32-6b9bb548f9-bzxk2" Oct 30 18:03:53.977269 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:53.977233 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw6cp\" (UniqueName: \"kubernetes.io/projected/3f13fc8d-3df1-4f51-816e-69f10c2a03cb-kube-api-access-gw6cp\") pod \"payload-pre-processing-e2e-worker-w4-419d32-6b9bb548f9-bzxk2\" (UID: \"3f13fc8d-3df1-4f51-816e-69f10c2a03cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-419d32-6b9bb548f9-bzxk2" Oct 30 18:03:53.977489 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:53.977363 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnllz\" (UniqueName: \"kubernetes.io/projected/c1b171c5-abf2-4fd2-9b7f-5d103f44f704-kube-api-access-pnllz\") pod \"payload-processing-e2e-worker-w4-419d32-6dff5c9996-jcbjp\" (UID: \"c1b171c5-abf2-4fd2-9b7f-5d103f44f704\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-419d32-6dff5c9996-jcbjp" Oct 30 18:03:53.977489 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:53.977389 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b171c5-abf2-4fd2-9b7f-5d103f44f704-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-419d32-6dff5c9996-jcbjp\" (UID: \"c1b171c5-abf2-4fd2-9b7f-5d103f44f704\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-419d32-6dff5c9996-jcbjp" Oct 30 18:03:53.977489 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:53.977454 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f13fc8d-3df1-4f51-816e-69f10c2a03cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-419d32-6b9bb548f9-bzxk2\" (UID: \"3f13fc8d-3df1-4f51-816e-69f10c2a03cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-419d32-6b9bb548f9-bzxk2" Oct 30 18:03:53.978149 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:53.978126 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f13fc8d-3df1-4f51-816e-69f10c2a03cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-419d32-6b9bb548f9-bzxk2\" (UID: \"3f13fc8d-3df1-4f51-816e-69f10c2a03cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-419d32-6b9bb548f9-bzxk2" Oct 30 18:03:53.978199 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:53.978134 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b171c5-abf2-4fd2-9b7f-5d103f44f704-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-419d32-6dff5c9996-jcbjp\" (UID: \"c1b171c5-abf2-4fd2-9b7f-5d103f44f704\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-419d32-6dff5c9996-jcbjp" Oct 30 18:03:54.000314 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:54.000283 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnllz\" (UniqueName: \"kubernetes.io/projected/c1b171c5-abf2-4fd2-9b7f-5d103f44f704-kube-api-access-pnllz\") pod \"payload-processing-e2e-worker-w4-419d32-6dff5c9996-jcbjp\" (UID: \"c1b171c5-abf2-4fd2-9b7f-5d103f44f704\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-419d32-6dff5c9996-jcbjp" Oct 30 18:03:54.000463 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:54.000288 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw6cp\" (UniqueName: \"kubernetes.io/projected/3f13fc8d-3df1-4f51-816e-69f10c2a03cb-kube-api-access-gw6cp\") pod \"payload-pre-processing-e2e-worker-w4-419d32-6b9bb548f9-bzxk2\" (UID: \"3f13fc8d-3df1-4f51-816e-69f10c2a03cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-419d32-6b9bb548f9-bzxk2" Oct 30 18:03:54.111690 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:54.111602 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-419d32-6dff5c9996-jcbjp" Oct 30 18:03:54.122858 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:54.122826 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-419d32-6b9bb548f9-bzxk2" Oct 30 18:03:54.422870 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:54.422838 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-419d32-6dff5c9996-jcbjp"] Oct 30 18:03:54.424712 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:03:54.424680 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b171c5_abf2_4fd2_9b7f_5d103f44f704.slice/crio-450a3b3f13ac05b3ff09efbe3def600495d31b1a8b328e15c6e73a2cd82d0132 WatchSource:0}: Error finding container 450a3b3f13ac05b3ff09efbe3def600495d31b1a8b328e15c6e73a2cd82d0132: Status 404 returned error can't find the container with id 450a3b3f13ac05b3ff09efbe3def600495d31b1a8b328e15c6e73a2cd82d0132 Oct 30 18:03:54.434166 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:54.434141 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-419d32-6b9bb548f9-bzxk2"] Oct 30 18:03:54.436348 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:03:54.436295 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f13fc8d_3df1_4f51_816e_69f10c2a03cb.slice/crio-429ad68aa945efe94999881158b02d77b2b6935f1579d7999739cb7bd528c644 WatchSource:0}: Error finding container 429ad68aa945efe94999881158b02d77b2b6935f1579d7999739cb7bd528c644: Status 404 returned error can't find the container with id 429ad68aa945efe94999881158b02d77b2b6935f1579d7999739cb7bd528c644 Oct 30 18:03:54.823745 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:54.823701 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-419d32-6dff5c9996-jcbjp" event={"ID":"c1b171c5-abf2-4fd2-9b7f-5d103f44f704","Type":"ContainerStarted","Data":"938d228428904ae56b628d23ca3afa1884eb384fea6ae20f0f67618c13456691"} Oct 30 18:03:54.824243 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:54.823776 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-419d32-6dff5c9996-jcbjp" event={"ID":"c1b171c5-abf2-4fd2-9b7f-5d103f44f704","Type":"ContainerStarted","Data":"450a3b3f13ac05b3ff09efbe3def600495d31b1a8b328e15c6e73a2cd82d0132"} Oct 30 18:03:54.824243 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:54.823941 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-419d32-6dff5c9996-jcbjp" Oct 30 18:03:54.825744 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:54.825721 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-419d32-6b9bb548f9-bzxk2" event={"ID":"3f13fc8d-3df1-4f51-816e-69f10c2a03cb","Type":"ContainerStarted","Data":"d303bae62f2934ffa406d88baf6d08feb06d2de54509ae21486a6273c664155e"} Oct 30 18:03:54.825862 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:54.825757 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-419d32-6b9bb548f9-bzxk2" event={"ID":"3f13fc8d-3df1-4f51-816e-69f10c2a03cb","Type":"ContainerStarted","Data":"429ad68aa945efe94999881158b02d77b2b6935f1579d7999739cb7bd528c644"} Oct 30 18:03:54.825862 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:54.825852 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-419d32-6b9bb548f9-bzxk2" Oct 30 18:03:54.872283 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:54.872228 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-419d32-6dff5c9996-jcbjp" podStartSLOduration=1.872212859 podStartE2EDuration="1.872212859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:54.871127505 +0000 UTC m=+687.358764545" watchObservedRunningTime="2026-10-30 18:03:54.872212859 +0000 UTC m=+687.359849893" Oct 30 18:03:54.918194 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:54.918145 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-419d32-6b9bb548f9-bzxk2" podStartSLOduration=1.918130218 podStartE2EDuration="1.918130218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:54.915071989 +0000 UTC m=+687.402709023" watchObservedRunningTime="2026-10-30 18:03:54.918130218 +0000 UTC m=+687.405767253" Oct 30 18:03:55.399559 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:55.399520 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg"] Oct 30 18:03:55.404820 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:55.404784 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg" Oct 30 18:03:55.425092 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:55.425055 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-419d32-serving-cert\"" Oct 30 18:03:55.435007 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:55.434976 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg"] Oct 30 18:03:55.498775 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:55.498735 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e62b028a-4ecb-46f8-9d6c-94bf236dc782-maas-api-tls\") pod \"maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg\" (UID: \"e62b028a-4ecb-46f8-9d6c-94bf236dc782\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg" Oct 30 18:03:55.498970 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:55.498940 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e62b028a-4ecb-46f8-9d6c-94bf236dc782-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg\" (UID: \"e62b028a-4ecb-46f8-9d6c-94bf236dc782\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg" Oct 30 18:03:55.499035 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:55.499025 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e62b028a-4ecb-46f8-9d6c-94bf236dc782-metrics-certs\") pod \"maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg\" (UID: \"e62b028a-4ecb-46f8-9d6c-94bf236dc782\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg" Oct 30 18:03:55.499091 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:55.499047 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg6lf\" (UniqueName: \"kubernetes.io/projected/e62b028a-4ecb-46f8-9d6c-94bf236dc782-kube-api-access-hg6lf\") pod \"maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg\" (UID: \"e62b028a-4ecb-46f8-9d6c-94bf236dc782\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg" Oct 30 18:03:55.600024 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:55.599988 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e62b028a-4ecb-46f8-9d6c-94bf236dc782-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg\" (UID: \"e62b028a-4ecb-46f8-9d6c-94bf236dc782\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg" Oct 30 18:03:55.600243 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:55.600062 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e62b028a-4ecb-46f8-9d6c-94bf236dc782-metrics-certs\") pod \"maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg\" (UID: \"e62b028a-4ecb-46f8-9d6c-94bf236dc782\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg" Oct 30 18:03:55.600243 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:55.600083 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg6lf\" (UniqueName: \"kubernetes.io/projected/e62b028a-4ecb-46f8-9d6c-94bf236dc782-kube-api-access-hg6lf\") pod \"maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg\" (UID: \"e62b028a-4ecb-46f8-9d6c-94bf236dc782\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg" Oct 30 18:03:55.600243 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:55.600145 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e62b028a-4ecb-46f8-9d6c-94bf236dc782-maas-api-tls\") pod \"maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg\" (UID: \"e62b028a-4ecb-46f8-9d6c-94bf236dc782\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg" Oct 30 18:03:55.600812 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:55.600788 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e62b028a-4ecb-46f8-9d6c-94bf236dc782-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg\" (UID: \"e62b028a-4ecb-46f8-9d6c-94bf236dc782\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg" Oct 30 18:03:55.603474 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:55.603443 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e62b028a-4ecb-46f8-9d6c-94bf236dc782-maas-api-tls\") pod \"maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg\" (UID: \"e62b028a-4ecb-46f8-9d6c-94bf236dc782\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg" Oct 30 18:03:55.604687 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:55.604664 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e62b028a-4ecb-46f8-9d6c-94bf236dc782-metrics-certs\") pod \"maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg\" (UID: \"e62b028a-4ecb-46f8-9d6c-94bf236dc782\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg" Oct 30 18:03:55.628403 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:55.628371 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg6lf\" (UniqueName: \"kubernetes.io/projected/e62b028a-4ecb-46f8-9d6c-94bf236dc782-kube-api-access-hg6lf\") pod \"maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg\" (UID: \"e62b028a-4ecb-46f8-9d6c-94bf236dc782\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg" Oct 30 18:03:55.725537 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:55.725507 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg" Oct 30 18:03:56.080853 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:56.080824 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg"] Oct 30 18:03:56.083124 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:03:56.083091 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode62b028a_4ecb_46f8_9d6c_94bf236dc782.slice/crio-ed19c24b68aa163cd7e3dbb5460461a3486bef7d2435eb6fd36a00b7aebe3a8a WatchSource:0}: Error finding container ed19c24b68aa163cd7e3dbb5460461a3486bef7d2435eb6fd36a00b7aebe3a8a: Status 404 returned error can't find the container with id ed19c24b68aa163cd7e3dbb5460461a3486bef7d2435eb6fd36a00b7aebe3a8a Oct 30 18:03:56.925726 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:56.925631 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg" event={"ID":"e62b028a-4ecb-46f8-9d6c-94bf236dc782","Type":"ContainerStarted","Data":"98115b037cfee465085dfba99e933e44ef8c1d9dff7ca6329f789a07af8f720d"} Oct 30 18:03:56.925726 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:56.925685 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg" event={"ID":"e62b028a-4ecb-46f8-9d6c-94bf236dc782","Type":"ContainerStarted","Data":"ed19c24b68aa163cd7e3dbb5460461a3486bef7d2435eb6fd36a00b7aebe3a8a"} Oct 30 18:03:56.925966 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:56.925769 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg" Oct 30 18:03:56.969460 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:56.969397 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg" podStartSLOduration=1.380899091 podStartE2EDuration="1.969376882s" totalImagesPullingTime="588.477791ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:56.966650461 +0000 UTC m=+689.454287498" watchObservedRunningTime="2026-10-30 18:03:56.969376882 +0000 UTC m=+689.457013920" Oct 30 18:03:58.821370 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:03:58.821307 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr" Oct 30 18:04:00.387728 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:04:00.387689 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:04:01.804792 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:01.804759 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6e8fca-79548bc54c-5nw4l" Oct 30 18:04:01.805308 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:01.804863 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-6e8fca-7df8c6d488-xwlc2" Oct 30 18:04:02.464380 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:02.464310 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a9a40825-7f544d65c4-gclds"] Oct 30 18:04:02.498644 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:02.498607 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a9a40825-84767fb78c-2fk5b"] Oct 30 18:04:02.933669 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:02.933617 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg" podUID="e62b028a-4ecb-46f8-9d6c-94bf236dc782" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.77:8443/health\": dial tcp 10.134.0.77:8443: connect: connection refused" Oct 30 18:04:04.176022 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:04.175985 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b424a5-7d97cc8c87-vvjcw"] Oct 30 18:04:04.182207 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:04.182184 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b424a5-7d97cc8c87-vvjcw" Oct 30 18:04:04.200415 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:04.200391 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-b424a5\"" Oct 30 18:04:04.211549 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:04.211511 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-b424a5-dockercfg-4lxgl\"" Oct 30 18:04:04.216107 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:04.216081 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b424a5-7d97cc8c87-vvjcw"] Oct 30 18:04:04.281083 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:04.281046 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b424a5-b698d4bb8-vvqmn"] Oct 30 18:04:04.286749 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:04.286713 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b424a5-b698d4bb8-vvqmn" Oct 30 18:04:04.302585 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:04.302229 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b424a5-b698d4bb8-vvqmn"] Oct 30 18:04:04.316179 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:04.316148 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwl6q\" (UniqueName: \"kubernetes.io/projected/1e050532-86d8-4954-ae7a-b0fcd484ad24-kube-api-access-pwl6q\") pod \"payload-pre-processing-e2e-worker-w1-b424a5-7d97cc8c87-vvjcw\" (UID: \"1e050532-86d8-4954-ae7a-b0fcd484ad24\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b424a5-7d97cc8c87-vvjcw" Oct 30 18:04:04.316387 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:04.316366 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e050532-86d8-4954-ae7a-b0fcd484ad24-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-b424a5-7d97cc8c87-vvjcw\" (UID: \"1e050532-86d8-4954-ae7a-b0fcd484ad24\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b424a5-7d97cc8c87-vvjcw" Oct 30 18:04:04.417286 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:04.417251 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61033891-2264-449f-9a4b-d5b775d51ca6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b424a5-b698d4bb8-vvqmn\" (UID: \"61033891-2264-449f-9a4b-d5b775d51ca6\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b424a5-b698d4bb8-vvqmn" Oct 30 18:04:04.417497 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:04.417358 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e050532-86d8-4954-ae7a-b0fcd484ad24-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-b424a5-7d97cc8c87-vvjcw\" (UID: \"1e050532-86d8-4954-ae7a-b0fcd484ad24\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b424a5-7d97cc8c87-vvjcw" Oct 30 18:04:04.417497 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:04.417426 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc8pg\" (UniqueName: \"kubernetes.io/projected/61033891-2264-449f-9a4b-d5b775d51ca6-kube-api-access-mc8pg\") pod \"payload-processing-e2e-worker-w1-b424a5-b698d4bb8-vvqmn\" (UID: \"61033891-2264-449f-9a4b-d5b775d51ca6\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b424a5-b698d4bb8-vvqmn" Oct 30 18:04:04.417497 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:04.417475 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwl6q\" (UniqueName: \"kubernetes.io/projected/1e050532-86d8-4954-ae7a-b0fcd484ad24-kube-api-access-pwl6q\") pod \"payload-pre-processing-e2e-worker-w1-b424a5-7d97cc8c87-vvjcw\" (UID: \"1e050532-86d8-4954-ae7a-b0fcd484ad24\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b424a5-7d97cc8c87-vvjcw" Oct 30 18:04:04.418205 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:04.418182 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e050532-86d8-4954-ae7a-b0fcd484ad24-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-b424a5-7d97cc8c87-vvjcw\" (UID: \"1e050532-86d8-4954-ae7a-b0fcd484ad24\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b424a5-7d97cc8c87-vvjcw" Oct 30 18:04:04.441395 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:04.441299 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwl6q\" (UniqueName: \"kubernetes.io/projected/1e050532-86d8-4954-ae7a-b0fcd484ad24-kube-api-access-pwl6q\") pod \"payload-pre-processing-e2e-worker-w1-b424a5-7d97cc8c87-vvjcw\" (UID: \"1e050532-86d8-4954-ae7a-b0fcd484ad24\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b424a5-7d97cc8c87-vvjcw" Oct 30 18:04:04.496920 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:04.496889 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b424a5-7d97cc8c87-vvjcw" Oct 30 18:04:04.518604 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:04.518576 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61033891-2264-449f-9a4b-d5b775d51ca6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b424a5-b698d4bb8-vvqmn\" (UID: \"61033891-2264-449f-9a4b-d5b775d51ca6\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b424a5-b698d4bb8-vvqmn" Oct 30 18:04:04.518715 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:04.518674 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc8pg\" (UniqueName: \"kubernetes.io/projected/61033891-2264-449f-9a4b-d5b775d51ca6-kube-api-access-mc8pg\") pod \"payload-processing-e2e-worker-w1-b424a5-b698d4bb8-vvqmn\" (UID: \"61033891-2264-449f-9a4b-d5b775d51ca6\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b424a5-b698d4bb8-vvqmn" Oct 30 18:04:04.519307 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:04.519279 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61033891-2264-449f-9a4b-d5b775d51ca6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b424a5-b698d4bb8-vvqmn\" (UID: \"61033891-2264-449f-9a4b-d5b775d51ca6\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b424a5-b698d4bb8-vvqmn" Oct 30 18:04:04.541995 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:04.541913 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc8pg\" (UniqueName: \"kubernetes.io/projected/61033891-2264-449f-9a4b-d5b775d51ca6-kube-api-access-mc8pg\") pod \"payload-processing-e2e-worker-w1-b424a5-b698d4bb8-vvqmn\" (UID: \"61033891-2264-449f-9a4b-d5b775d51ca6\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b424a5-b698d4bb8-vvqmn" Oct 30 18:04:04.607708 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:04.607673 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b424a5-b698d4bb8-vvqmn" Oct 30 18:04:04.805772 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:04.805728 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b424a5-7d97cc8c87-vvjcw"] Oct 30 18:04:04.809011 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:04:04.808982 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e050532_86d8_4954_ae7a_b0fcd484ad24.slice/crio-8459fc63bd7e472105839c9984b8e3ef873cec5df7bbe3fc2cca5ee89035399e WatchSource:0}: Error finding container 8459fc63bd7e472105839c9984b8e3ef873cec5df7bbe3fc2cca5ee89035399e: Status 404 returned error can't find the container with id 8459fc63bd7e472105839c9984b8e3ef873cec5df7bbe3fc2cca5ee89035399e Oct 30 18:04:04.922787 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:04.922767 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b424a5-b698d4bb8-vvqmn"] Oct 30 18:04:04.924583 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:04:04.924553 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61033891_2264_449f_9a4b_d5b775d51ca6.slice/crio-f1b44fc6d47ff9c9a7198ed33bbe077e8e13b075b32e0afddd26a09be6374eb1 WatchSource:0}: Error finding container f1b44fc6d47ff9c9a7198ed33bbe077e8e13b075b32e0afddd26a09be6374eb1: Status 404 returned error can't find the container with id f1b44fc6d47ff9c9a7198ed33bbe077e8e13b075b32e0afddd26a09be6374eb1 Oct 30 18:04:04.978916 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:04.978887 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b424a5-b698d4bb8-vvqmn" event={"ID":"61033891-2264-449f-9a4b-d5b775d51ca6","Type":"ContainerStarted","Data":"f1b44fc6d47ff9c9a7198ed33bbe077e8e13b075b32e0afddd26a09be6374eb1"} Oct 30 18:04:04.981198 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:04.981175 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b424a5-7d97cc8c87-vvjcw" event={"ID":"1e050532-86d8-4954-ae7a-b0fcd484ad24","Type":"ContainerStarted","Data":"eeda27f88a2295a5daf43a0093fc91d5b02d804fd8aec3821f95168389ec5491"} Oct 30 18:04:04.981335 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:04.981227 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b424a5-7d97cc8c87-vvjcw" event={"ID":"1e050532-86d8-4954-ae7a-b0fcd484ad24","Type":"ContainerStarted","Data":"8459fc63bd7e472105839c9984b8e3ef873cec5df7bbe3fc2cca5ee89035399e"} Oct 30 18:04:04.981419 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:04.981355 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b424a5-7d97cc8c87-vvjcw" Oct 30 18:04:05.022575 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:05.022510 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b424a5-7d97cc8c87-vvjcw" podStartSLOduration=1.022481624 podStartE2EDuration="1.022481624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:04:05.020111724 +0000 UTC m=+697.507748757" watchObservedRunningTime="2026-10-30 18:04:05.022481624 +0000 UTC m=+697.510118660" Oct 30 18:04:05.928425 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:05.928388 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-419d32-6dff5c9996-jcbjp" Oct 30 18:04:05.928847 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:05.928763 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-419d32-6b9bb548f9-bzxk2" Oct 30 18:04:05.990954 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:05.990924 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b424a5-b698d4bb8-vvqmn" event={"ID":"61033891-2264-449f-9a4b-d5b775d51ca6","Type":"ContainerStarted","Data":"6876f31cc94d72383d5346128cd01f57e4cfd518824d39c00be506c9c15a4bc8"} Oct 30 18:04:06.046997 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:06.046943 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-b424a5-b698d4bb8-vvqmn" podStartSLOduration=2.046928157 podStartE2EDuration="2.046928157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:04:06.045020165 +0000 UTC m=+698.532657199" watchObservedRunningTime="2026-10-30 18:04:06.046928157 +0000 UTC m=+698.534565190" Oct 30 18:04:06.999446 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:06.999414 4455 generic.go:356] "Generic (PLEG): container finished" podID="e62b028a-4ecb-46f8-9d6c-94bf236dc782" containerID="98115b037cfee465085dfba99e933e44ef8c1d9dff7ca6329f789a07af8f720d" exitCode=1 Oct 30 18:04:06.999819 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:06.999489 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg" event={"ID":"e62b028a-4ecb-46f8-9d6c-94bf236dc782","Type":"ContainerDied","Data":"98115b037cfee465085dfba99e933e44ef8c1d9dff7ca6329f789a07af8f720d"} Oct 30 18:04:06.999819 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:06.999656 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-b424a5-b698d4bb8-vvqmn" Oct 30 18:04:06.999919 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:06.999905 4455 scope.go:122] "RemoveContainer" containerID="98115b037cfee465085dfba99e933e44ef8c1d9dff7ca6329f789a07af8f720d" Oct 30 18:04:07.465260 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:07.465219 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a9a40825-7f544d65c4-gclds" podUID="8ca4b9f7-4a87-474c-9469-118a08eb62ca" containerName="payload-processing" containerID="cri-o://71cc57a842681896f3ba9411e93798e6ed144e641722420b8398a920db798502" gracePeriod=25 Oct 30 18:04:07.499802 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:07.499754 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9a40825-84767fb78c-2fk5b" podUID="315ffa4a-2726-439c-b387-d5d148c48524" containerName="payload-pre-processing" containerID="cri-o://cce0bb36e248c1887db96220b555c808f9d90e352767273db8bb2f42a2cce32f" gracePeriod=25 Oct 30 18:04:07.762495 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:07.762472 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a9a40825-7f544d65c4-gclds" Oct 30 18:04:07.785756 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:07.785734 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9a40825-84767fb78c-2fk5b" Oct 30 18:04:07.856407 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:07.856309 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ca4b9f7-4a87-474c-9469-118a08eb62ca-kube-api-access-6f79v\" (UniqueName: \"kubernetes.io/projected/8ca4b9f7-4a87-474c-9469-118a08eb62ca-kube-api-access-6f79v\") pod \"8ca4b9f7-4a87-474c-9469-118a08eb62ca\" (UID: \"8ca4b9f7-4a87-474c-9469-118a08eb62ca\") " Oct 30 18:04:07.856407 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:07.856393 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/315ffa4a-2726-439c-b387-d5d148c48524-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/315ffa4a-2726-439c-b387-d5d148c48524-plugins-config-volume\") pod \"315ffa4a-2726-439c-b387-d5d148c48524\" (UID: \"315ffa4a-2726-439c-b387-d5d148c48524\") " Oct 30 18:04:07.856634 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:07.856486 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ca4b9f7-4a87-474c-9469-118a08eb62ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca4b9f7-4a87-474c-9469-118a08eb62ca-plugins-config-volume\") pod \"8ca4b9f7-4a87-474c-9469-118a08eb62ca\" (UID: \"8ca4b9f7-4a87-474c-9469-118a08eb62ca\") " Oct 30 18:04:07.856634 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:07.856597 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/315ffa4a-2726-439c-b387-d5d148c48524-kube-api-access-7wv9k\" (UniqueName: \"kubernetes.io/projected/315ffa4a-2726-439c-b387-d5d148c48524-kube-api-access-7wv9k\") pod \"315ffa4a-2726-439c-b387-d5d148c48524\" (UID: \"315ffa4a-2726-439c-b387-d5d148c48524\") " Oct 30 18:04:07.856868 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:07.856833 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/315ffa4a-2726-439c-b387-d5d148c48524-plugins-config-volume" pod "315ffa4a-2726-439c-b387-d5d148c48524" (UID: "315ffa4a-2726-439c-b387-d5d148c48524"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:04:07.856952 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:07.856925 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ca4b9f7-4a87-474c-9469-118a08eb62ca-plugins-config-volume" pod "8ca4b9f7-4a87-474c-9469-118a08eb62ca" (UID: "8ca4b9f7-4a87-474c-9469-118a08eb62ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:04:07.857071 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:07.857040 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca4b9f7-4a87-474c-9469-118a08eb62ca-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:04:07.857126 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:07.857069 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/315ffa4a-2726-439c-b387-d5d148c48524-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:04:07.859158 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:07.859136 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ca4b9f7-4a87-474c-9469-118a08eb62ca-kube-api-access-6f79v" pod "8ca4b9f7-4a87-474c-9469-118a08eb62ca" (UID: "8ca4b9f7-4a87-474c-9469-118a08eb62ca"). InnerVolumeSpecName "kube-api-access-6f79v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:04:07.859265 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:07.859246 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/315ffa4a-2726-439c-b387-d5d148c48524-kube-api-access-7wv9k" pod "315ffa4a-2726-439c-b387-d5d148c48524" (UID: "315ffa4a-2726-439c-b387-d5d148c48524"). InnerVolumeSpecName "kube-api-access-7wv9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:04:07.957635 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:07.957605 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wv9k\" (UniqueName: \"kubernetes.io/projected/315ffa4a-2726-439c-b387-d5d148c48524-kube-api-access-7wv9k\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:04:07.957635 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:07.957629 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f79v\" (UniqueName: \"kubernetes.io/projected/8ca4b9f7-4a87-474c-9469-118a08eb62ca-kube-api-access-6f79v\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:04:08.007663 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:08.007630 4455 generic.go:356] "Generic (PLEG): container finished" podID="315ffa4a-2726-439c-b387-d5d148c48524" containerID="cce0bb36e248c1887db96220b555c808f9d90e352767273db8bb2f42a2cce32f" exitCode=0 Oct 30 18:04:08.008150 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:08.007680 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9a40825-84767fb78c-2fk5b" Oct 30 18:04:08.008150 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:08.007724 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9a40825-84767fb78c-2fk5b" event={"ID":"315ffa4a-2726-439c-b387-d5d148c48524","Type":"ContainerDied","Data":"cce0bb36e248c1887db96220b555c808f9d90e352767273db8bb2f42a2cce32f"} Oct 30 18:04:08.008150 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:08.007799 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a9a40825-84767fb78c-2fk5b" event={"ID":"315ffa4a-2726-439c-b387-d5d148c48524","Type":"ContainerDied","Data":"d1e47502f7d49582b4005733ad71a4fdfae28bc3d2c127c48394bd10f9dfe14e"} Oct 30 18:04:08.008150 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:08.007828 4455 scope.go:122] "RemoveContainer" containerID="cce0bb36e248c1887db96220b555c808f9d90e352767273db8bb2f42a2cce32f" Oct 30 18:04:08.009926 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:08.009892 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg" event={"ID":"e62b028a-4ecb-46f8-9d6c-94bf236dc782","Type":"ContainerStarted","Data":"ef81a43bef31fccac078983d3c32dc95a61bcafc616372bd7d12f44baa117679"} Oct 30 18:04:08.010114 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:08.010098 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg" Oct 30 18:04:08.011388 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:08.011367 4455 generic.go:356] "Generic (PLEG): container finished" podID="8ca4b9f7-4a87-474c-9469-118a08eb62ca" containerID="71cc57a842681896f3ba9411e93798e6ed144e641722420b8398a920db798502" exitCode=0 Oct 30 18:04:08.011564 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:08.011419 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a9a40825-7f544d65c4-gclds" Oct 30 18:04:08.011564 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:08.011455 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a9a40825-7f544d65c4-gclds" event={"ID":"8ca4b9f7-4a87-474c-9469-118a08eb62ca","Type":"ContainerDied","Data":"71cc57a842681896f3ba9411e93798e6ed144e641722420b8398a920db798502"} Oct 30 18:04:08.011564 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:08.011496 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a9a40825-7f544d65c4-gclds" event={"ID":"8ca4b9f7-4a87-474c-9469-118a08eb62ca","Type":"ContainerDied","Data":"9c21a31b351a9c514a008b40069a70af426c2bdd1dcca0e2ae86ff211e40669c"} Oct 30 18:04:08.020868 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:08.020845 4455 scope.go:122] "RemoveContainer" containerID="cce0bb36e248c1887db96220b555c808f9d90e352767273db8bb2f42a2cce32f" Oct 30 18:04:08.021176 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:04:08.021147 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cce0bb36e248c1887db96220b555c808f9d90e352767273db8bb2f42a2cce32f\": container with ID starting with cce0bb36e248c1887db96220b555c808f9d90e352767273db8bb2f42a2cce32f not found: ID does not exist" containerID="cce0bb36e248c1887db96220b555c808f9d90e352767273db8bb2f42a2cce32f" Oct 30 18:04:08.021289 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:08.021184 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce0bb36e248c1887db96220b555c808f9d90e352767273db8bb2f42a2cce32f"} err="failed to get container status \"cce0bb36e248c1887db96220b555c808f9d90e352767273db8bb2f42a2cce32f\": rpc error: code = NotFound desc = could not find container \"cce0bb36e248c1887db96220b555c808f9d90e352767273db8bb2f42a2cce32f\": container with ID starting with cce0bb36e248c1887db96220b555c808f9d90e352767273db8bb2f42a2cce32f not found: ID does not exist" Oct 30 18:04:08.021289 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:08.021207 4455 scope.go:122] "RemoveContainer" containerID="71cc57a842681896f3ba9411e93798e6ed144e641722420b8398a920db798502" Oct 30 18:04:08.031898 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:08.031879 4455 scope.go:122] "RemoveContainer" containerID="71cc57a842681896f3ba9411e93798e6ed144e641722420b8398a920db798502" Oct 30 18:04:08.032158 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:04:08.032140 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71cc57a842681896f3ba9411e93798e6ed144e641722420b8398a920db798502\": container with ID starting with 71cc57a842681896f3ba9411e93798e6ed144e641722420b8398a920db798502 not found: ID does not exist" containerID="71cc57a842681896f3ba9411e93798e6ed144e641722420b8398a920db798502" Oct 30 18:04:08.032241 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:08.032166 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71cc57a842681896f3ba9411e93798e6ed144e641722420b8398a920db798502"} err="failed to get container status \"71cc57a842681896f3ba9411e93798e6ed144e641722420b8398a920db798502\": rpc error: code = NotFound desc = could not find container \"71cc57a842681896f3ba9411e93798e6ed144e641722420b8398a920db798502\": container with ID starting with 71cc57a842681896f3ba9411e93798e6ed144e641722420b8398a920db798502 not found: ID does not exist" Oct 30 18:04:08.071994 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:08.071971 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a9a40825-84767fb78c-2fk5b"] Oct 30 18:04:08.076865 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:08.076843 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a9a40825-84767fb78c-2fk5b"] Oct 30 18:04:08.158139 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:08.158053 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a9a40825-7f544d65c4-gclds"] Oct 30 18:04:08.164052 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:08.164025 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a9a40825-7f544d65c4-gclds"] Oct 30 18:04:09.119210 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.119172 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3cc458-779ff4f8cf-lt8qr"] Oct 30 18:04:09.119762 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.119746 4455 cpu_manager.go:414] "containerMap: removing container" podUID="315ffa4a-2726-439c-b387-d5d148c48524" containerName="payload-pre-processing" Oct 30 18:04:09.119807 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.119764 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="315ffa4a-2726-439c-b387-d5d148c48524" containerName="payload-pre-processing" Oct 30 18:04:09.119807 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.119805 4455 cpu_manager.go:414] "containerMap: removing container" podUID="8ca4b9f7-4a87-474c-9469-118a08eb62ca" containerName="payload-processing" Oct 30 18:04:09.119880 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.119811 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ca4b9f7-4a87-474c-9469-118a08eb62ca" containerName="payload-processing" Oct 30 18:04:09.119910 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.119889 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ca4b9f7-4a87-474c-9469-118a08eb62ca" containerName="payload-processing" Oct 30 18:04:09.119910 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.119902 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="315ffa4a-2726-439c-b387-d5d148c48524" containerName="payload-pre-processing" Oct 30 18:04:09.124734 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.124715 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3cc458-779ff4f8cf-lt8qr" Oct 30 18:04:09.139709 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.139688 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-3cc458-dockercfg-wr65c\"" Oct 30 18:04:09.151434 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.151410 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-3cc458\"" Oct 30 18:04:09.154912 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.154885 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3cc458-779ff4f8cf-lt8qr"] Oct 30 18:04:09.190886 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.190856 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3cc458-99565d668-p2qbm"] Oct 30 18:04:09.195758 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.195737 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc458-99565d668-p2qbm" Oct 30 18:04:09.214911 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.214880 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3cc458-99565d668-p2qbm"] Oct 30 18:04:09.231253 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.231131 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n"] Oct 30 18:04:09.237155 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.236441 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n" Oct 30 18:04:09.271393 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.271361 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-b424a5-serving-cert\"" Oct 30 18:04:09.275058 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.275024 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b13dbfe-f5db-46e6-810e-9d4ac231c94c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-3cc458-779ff4f8cf-lt8qr\" (UID: \"2b13dbfe-f5db-46e6-810e-9d4ac231c94c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3cc458-779ff4f8cf-lt8qr" Oct 30 18:04:09.275830 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.275803 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rsrm\" (UniqueName: \"kubernetes.io/projected/c63830ed-816d-4e24-9900-7ee260bf622f-kube-api-access-5rsrm\") pod \"payload-processing-e2e-worker-w3-3cc458-99565d668-p2qbm\" (UID: \"c63830ed-816d-4e24-9900-7ee260bf622f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc458-99565d668-p2qbm" Oct 30 18:04:09.276085 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.276065 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c63830ed-816d-4e24-9900-7ee260bf622f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-3cc458-99565d668-p2qbm\" (UID: \"c63830ed-816d-4e24-9900-7ee260bf622f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc458-99565d668-p2qbm" Oct 30 18:04:09.276312 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.276282 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khnrs\" (UniqueName: \"kubernetes.io/projected/2b13dbfe-f5db-46e6-810e-9d4ac231c94c-kube-api-access-khnrs\") pod \"payload-pre-processing-e2e-worker-w3-3cc458-779ff4f8cf-lt8qr\" (UID: \"2b13dbfe-f5db-46e6-810e-9d4ac231c94c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3cc458-779ff4f8cf-lt8qr" Oct 30 18:04:09.298304 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.298277 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n"] Oct 30 18:04:09.377229 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.377148 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b13dbfe-f5db-46e6-810e-9d4ac231c94c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-3cc458-779ff4f8cf-lt8qr\" (UID: \"2b13dbfe-f5db-46e6-810e-9d4ac231c94c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3cc458-779ff4f8cf-lt8qr" Oct 30 18:04:09.377229 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.377195 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rsrm\" (UniqueName: \"kubernetes.io/projected/c63830ed-816d-4e24-9900-7ee260bf622f-kube-api-access-5rsrm\") pod \"payload-processing-e2e-worker-w3-3cc458-99565d668-p2qbm\" (UID: \"c63830ed-816d-4e24-9900-7ee260bf622f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc458-99565d668-p2qbm" Oct 30 18:04:09.377458 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.377226 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbtlw\" (UniqueName: \"kubernetes.io/projected/dbdbb8b6-80ee-4abe-990e-72e4add7e77b-kube-api-access-rbtlw\") pod \"maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n\" (UID: \"dbdbb8b6-80ee-4abe-990e-72e4add7e77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n" Oct 30 18:04:09.377458 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.377385 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbdbb8b6-80ee-4abe-990e-72e4add7e77b-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n\" (UID: \"dbdbb8b6-80ee-4abe-990e-72e4add7e77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n" Oct 30 18:04:09.377458 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.377432 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c63830ed-816d-4e24-9900-7ee260bf622f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-3cc458-99565d668-p2qbm\" (UID: \"c63830ed-816d-4e24-9900-7ee260bf622f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc458-99565d668-p2qbm" Oct 30 18:04:09.377613 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.377482 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbdbb8b6-80ee-4abe-990e-72e4add7e77b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n\" (UID: \"dbdbb8b6-80ee-4abe-990e-72e4add7e77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n" Oct 30 18:04:09.377669 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.377616 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbdbb8b6-80ee-4abe-990e-72e4add7e77b-metrics-certs\") pod \"maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n\" (UID: \"dbdbb8b6-80ee-4abe-990e-72e4add7e77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n" Oct 30 18:04:09.377723 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.377661 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khnrs\" (UniqueName: \"kubernetes.io/projected/2b13dbfe-f5db-46e6-810e-9d4ac231c94c-kube-api-access-khnrs\") pod \"payload-pre-processing-e2e-worker-w3-3cc458-779ff4f8cf-lt8qr\" (UID: \"2b13dbfe-f5db-46e6-810e-9d4ac231c94c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3cc458-779ff4f8cf-lt8qr" Oct 30 18:04:09.378076 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.378050 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b13dbfe-f5db-46e6-810e-9d4ac231c94c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-3cc458-779ff4f8cf-lt8qr\" (UID: \"2b13dbfe-f5db-46e6-810e-9d4ac231c94c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3cc458-779ff4f8cf-lt8qr" Oct 30 18:04:09.378180 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.378165 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c63830ed-816d-4e24-9900-7ee260bf622f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-3cc458-99565d668-p2qbm\" (UID: \"c63830ed-816d-4e24-9900-7ee260bf622f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc458-99565d668-p2qbm" Oct 30 18:04:09.413492 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.413459 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rsrm\" (UniqueName: \"kubernetes.io/projected/c63830ed-816d-4e24-9900-7ee260bf622f-kube-api-access-5rsrm\") pod \"payload-processing-e2e-worker-w3-3cc458-99565d668-p2qbm\" (UID: \"c63830ed-816d-4e24-9900-7ee260bf622f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc458-99565d668-p2qbm" Oct 30 18:04:09.413639 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.413616 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khnrs\" (UniqueName: \"kubernetes.io/projected/2b13dbfe-f5db-46e6-810e-9d4ac231c94c-kube-api-access-khnrs\") pod \"payload-pre-processing-e2e-worker-w3-3cc458-779ff4f8cf-lt8qr\" (UID: \"2b13dbfe-f5db-46e6-810e-9d4ac231c94c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3cc458-779ff4f8cf-lt8qr" Oct 30 18:04:09.438569 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.438550 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3cc458-779ff4f8cf-lt8qr" Oct 30 18:04:09.486380 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.486308 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbdbb8b6-80ee-4abe-990e-72e4add7e77b-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n\" (UID: \"dbdbb8b6-80ee-4abe-990e-72e4add7e77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n" Oct 30 18:04:09.486580 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.486444 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbdbb8b6-80ee-4abe-990e-72e4add7e77b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n\" (UID: \"dbdbb8b6-80ee-4abe-990e-72e4add7e77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n" Oct 30 18:04:09.486668 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.486649 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbdbb8b6-80ee-4abe-990e-72e4add7e77b-metrics-certs\") pod \"maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n\" (UID: \"dbdbb8b6-80ee-4abe-990e-72e4add7e77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n" Oct 30 18:04:09.486759 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.486741 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbtlw\" (UniqueName: \"kubernetes.io/projected/dbdbb8b6-80ee-4abe-990e-72e4add7e77b-kube-api-access-rbtlw\") pod \"maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n\" (UID: \"dbdbb8b6-80ee-4abe-990e-72e4add7e77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n" Oct 30 18:04:09.491781 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.491752 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbdbb8b6-80ee-4abe-990e-72e4add7e77b-metrics-certs\") pod \"maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n\" (UID: \"dbdbb8b6-80ee-4abe-990e-72e4add7e77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n" Oct 30 18:04:09.493724 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.493657 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbdbb8b6-80ee-4abe-990e-72e4add7e77b-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n\" (UID: \"dbdbb8b6-80ee-4abe-990e-72e4add7e77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n" Oct 30 18:04:09.496665 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.496618 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbdbb8b6-80ee-4abe-990e-72e4add7e77b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n\" (UID: \"dbdbb8b6-80ee-4abe-990e-72e4add7e77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n" Oct 30 18:04:09.514598 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.514544 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc458-99565d668-p2qbm" Oct 30 18:04:09.526628 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.524981 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbtlw\" (UniqueName: \"kubernetes.io/projected/dbdbb8b6-80ee-4abe-990e-72e4add7e77b-kube-api-access-rbtlw\") pod \"maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n\" (UID: \"dbdbb8b6-80ee-4abe-990e-72e4add7e77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n" Oct 30 18:04:09.599387 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.598990 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n" Oct 30 18:04:09.641455 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.641249 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="315ffa4a-2726-439c-b387-d5d148c48524" path="/var/lib/kubelet/pods/315ffa4a-2726-439c-b387-d5d148c48524/volumes" Oct 30 18:04:09.644084 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.643375 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ca4b9f7-4a87-474c-9469-118a08eb62ca" path="/var/lib/kubelet/pods/8ca4b9f7-4a87-474c-9469-118a08eb62ca/volumes" Oct 30 18:04:09.755870 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.755841 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3cc458-779ff4f8cf-lt8qr"] Oct 30 18:04:09.757013 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:04:09.756986 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b13dbfe_f5db_46e6_810e_9d4ac231c94c.slice/crio-2740a3c47cf21fdc091893f2b137d22e8da4e3a92ce92149871138c2b2bc5da9 WatchSource:0}: Error finding container 2740a3c47cf21fdc091893f2b137d22e8da4e3a92ce92149871138c2b2bc5da9: Status 404 returned error can't find the container with id 2740a3c47cf21fdc091893f2b137d22e8da4e3a92ce92149871138c2b2bc5da9 Oct 30 18:04:09.840746 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.840716 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3cc458-99565d668-p2qbm"] Oct 30 18:04:09.841787 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:04:09.841764 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc63830ed_816d_4e24_9900_7ee260bf622f.slice/crio-be3922a24dd3acc13abead6fb78648ee594d834ead59fe02eeeec86ce235ac3b WatchSource:0}: Error finding container be3922a24dd3acc13abead6fb78648ee594d834ead59fe02eeeec86ce235ac3b: Status 404 returned error can't find the container with id be3922a24dd3acc13abead6fb78648ee594d834ead59fe02eeeec86ce235ac3b Oct 30 18:04:09.919069 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:09.919042 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n"] Oct 30 18:04:09.920967 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:04:09.920938 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbdbb8b6_80ee_4abe_990e_72e4add7e77b.slice/crio-206be537d45ae691981b5915fb6b83257f06c29dd5b37e404de013159cc99eb1 WatchSource:0}: Error finding container 206be537d45ae691981b5915fb6b83257f06c29dd5b37e404de013159cc99eb1: Status 404 returned error can't find the container with id 206be537d45ae691981b5915fb6b83257f06c29dd5b37e404de013159cc99eb1 Oct 30 18:04:10.029408 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:10.029362 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3cc458-779ff4f8cf-lt8qr" event={"ID":"2b13dbfe-f5db-46e6-810e-9d4ac231c94c","Type":"ContainerStarted","Data":"e95c10fcd9389a7ec3750097dcf42557ad184a631ad5c3a284dedfb38fdd2f2f"} Oct 30 18:04:10.029590 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:10.029436 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3cc458-779ff4f8cf-lt8qr" event={"ID":"2b13dbfe-f5db-46e6-810e-9d4ac231c94c","Type":"ContainerStarted","Data":"2740a3c47cf21fdc091893f2b137d22e8da4e3a92ce92149871138c2b2bc5da9"} Oct 30 18:04:10.029590 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:10.029470 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3cc458-779ff4f8cf-lt8qr" Oct 30 18:04:10.031208 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:10.031181 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc458-99565d668-p2qbm" event={"ID":"c63830ed-816d-4e24-9900-7ee260bf622f","Type":"ContainerStarted","Data":"4e6092f2a06e7cc0ce2d302edda308824e8a51731a79701cb6e067a4345a6f4a"} Oct 30 18:04:10.031371 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:10.031233 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc458-99565d668-p2qbm" Oct 30 18:04:10.031371 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:10.031251 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc458-99565d668-p2qbm" event={"ID":"c63830ed-816d-4e24-9900-7ee260bf622f","Type":"ContainerStarted","Data":"be3922a24dd3acc13abead6fb78648ee594d834ead59fe02eeeec86ce235ac3b"} Oct 30 18:04:10.032559 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:10.032534 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n" event={"ID":"dbdbb8b6-80ee-4abe-990e-72e4add7e77b","Type":"ContainerStarted","Data":"206be537d45ae691981b5915fb6b83257f06c29dd5b37e404de013159cc99eb1"} Oct 30 18:04:10.072654 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:10.072539 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3cc458-779ff4f8cf-lt8qr" podStartSLOduration=1.072524245 podStartE2EDuration="1.072524245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:04:10.071381354 +0000 UTC m=+702.559018387" watchObservedRunningTime="2026-10-30 18:04:10.072524245 +0000 UTC m=+702.560161280" Oct 30 18:04:10.115713 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:10.115670 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc458-99565d668-p2qbm" podStartSLOduration=1.115656228 podStartE2EDuration="1.115656228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:04:10.11391107 +0000 UTC m=+702.601548103" watchObservedRunningTime="2026-10-30 18:04:10.115656228 +0000 UTC m=+702.603293262" Oct 30 18:04:11.040665 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:11.040634 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n" event={"ID":"dbdbb8b6-80ee-4abe-990e-72e4add7e77b","Type":"ContainerStarted","Data":"316a88c2717a8b365e51913c51d1d1935b2e763c6c265d44ed52ae48312f3001"} Oct 30 18:04:11.041110 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:11.040844 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n" Oct 30 18:04:11.082796 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:11.082743 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n" podStartSLOduration=1.5573975820000001 podStartE2EDuration="2.082729773s" totalImagesPullingTime="525.332191ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:04:11.081606608 +0000 UTC m=+703.569243641" watchObservedRunningTime="2026-10-30 18:04:11.082729773 +0000 UTC m=+703.570366806" Oct 30 18:04:14.024994 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:14.024967 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg" Oct 30 18:04:15.994164 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:15.994134 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b424a5-7d97cc8c87-vvjcw" Oct 30 18:04:17.052627 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:17.052599 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n" Oct 30 18:04:18.015771 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:18.015741 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-b424a5-b698d4bb8-vvqmn" Oct 30 18:04:18.673577 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:18.673538 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9zh44"] Oct 30 18:04:18.678217 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:18.678194 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9zh44" Oct 30 18:04:18.691357 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:18.691312 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p85j8\" (UniqueName: \"kubernetes.io/projected/c4fd1169-a9c3-4c6d-ba92-1a882c7309b0-kube-api-access-p85j8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9zh44\" (UID: \"c4fd1169-a9c3-4c6d-ba92-1a882c7309b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9zh44" Oct 30 18:04:18.691460 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:18.691421 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c4fd1169-a9c3-4c6d-ba92-1a882c7309b0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9zh44\" (UID: \"c4fd1169-a9c3-4c6d-ba92-1a882c7309b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9zh44" Oct 30 18:04:18.691508 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:18.691484 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c4fd1169-a9c3-4c6d-ba92-1a882c7309b0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9zh44\" (UID: \"c4fd1169-a9c3-4c6d-ba92-1a882c7309b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9zh44" Oct 30 18:04:18.691591 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:18.691555 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4fd1169-a9c3-4c6d-ba92-1a882c7309b0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9zh44\" (UID: \"c4fd1169-a9c3-4c6d-ba92-1a882c7309b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9zh44" Oct 30 18:04:18.691641 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:18.691606 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c4fd1169-a9c3-4c6d-ba92-1a882c7309b0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9zh44\" (UID: \"c4fd1169-a9c3-4c6d-ba92-1a882c7309b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9zh44" Oct 30 18:04:18.691641 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:18.691626 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c4fd1169-a9c3-4c6d-ba92-1a882c7309b0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9zh44\" (UID: \"c4fd1169-a9c3-4c6d-ba92-1a882c7309b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9zh44" Oct 30 18:04:18.691705 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:18.691645 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4fd1169-a9c3-4c6d-ba92-1a882c7309b0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9zh44\" (UID: \"c4fd1169-a9c3-4c6d-ba92-1a882c7309b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9zh44" Oct 30 18:04:18.693921 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:18.693902 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 18:04:18.694069 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:18.694055 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 18:04:18.704763 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:18.704696 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-44kjj\"" Oct 30 18:04:18.704763 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:18.704697 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 18:04:18.716255 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:18.716229 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9zh44"] Oct 30 18:04:18.792789 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:18.792747 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p85j8\" (UniqueName: \"kubernetes.io/projected/c4fd1169-a9c3-4c6d-ba92-1a882c7309b0-kube-api-access-p85j8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9zh44\" (UID: \"c4fd1169-a9c3-4c6d-ba92-1a882c7309b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9zh44" Oct 30 18:04:18.792999 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:18.792812 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c4fd1169-a9c3-4c6d-ba92-1a882c7309b0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9zh44\" (UID: \"c4fd1169-a9c3-4c6d-ba92-1a882c7309b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9zh44" Oct 30 18:04:18.792999 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:18.792867 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c4fd1169-a9c3-4c6d-ba92-1a882c7309b0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9zh44\" (UID: \"c4fd1169-a9c3-4c6d-ba92-1a882c7309b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9zh44" Oct 30 18:04:18.792999 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:18.792913 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4fd1169-a9c3-4c6d-ba92-1a882c7309b0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9zh44\" (UID: \"c4fd1169-a9c3-4c6d-ba92-1a882c7309b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9zh44" Oct 30 18:04:18.792999 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:18.792975 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c4fd1169-a9c3-4c6d-ba92-1a882c7309b0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9zh44\" (UID: \"c4fd1169-a9c3-4c6d-ba92-1a882c7309b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9zh44" Oct 30 18:04:18.793216 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:18.793004 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c4fd1169-a9c3-4c6d-ba92-1a882c7309b0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9zh44\" (UID: \"c4fd1169-a9c3-4c6d-ba92-1a882c7309b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9zh44" Oct 30 18:04:18.793216 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:18.793035 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4fd1169-a9c3-4c6d-ba92-1a882c7309b0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9zh44\" (UID: \"c4fd1169-a9c3-4c6d-ba92-1a882c7309b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9zh44" Oct 30 18:04:18.793755 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:18.793725 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c4fd1169-a9c3-4c6d-ba92-1a882c7309b0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9zh44\" (UID: \"c4fd1169-a9c3-4c6d-ba92-1a882c7309b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9zh44" Oct 30 18:04:18.796384 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:18.796355 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4fd1169-a9c3-4c6d-ba92-1a882c7309b0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9zh44\" (UID: \"c4fd1169-a9c3-4c6d-ba92-1a882c7309b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9zh44" Oct 30 18:04:18.796526 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:18.796504 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c4fd1169-a9c3-4c6d-ba92-1a882c7309b0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9zh44\" (UID: \"c4fd1169-a9c3-4c6d-ba92-1a882c7309b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9zh44" Oct 30 18:04:18.796711 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:18.796690 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c4fd1169-a9c3-4c6d-ba92-1a882c7309b0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9zh44\" (UID: \"c4fd1169-a9c3-4c6d-ba92-1a882c7309b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9zh44" Oct 30 18:04:18.796821 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:18.796806 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c4fd1169-a9c3-4c6d-ba92-1a882c7309b0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9zh44\" (UID: \"c4fd1169-a9c3-4c6d-ba92-1a882c7309b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9zh44" Oct 30 18:04:18.823018 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:18.822992 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4fd1169-a9c3-4c6d-ba92-1a882c7309b0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9zh44\" (UID: \"c4fd1169-a9c3-4c6d-ba92-1a882c7309b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9zh44" Oct 30 18:04:18.823168 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:18.823145 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p85j8\" (UniqueName: \"kubernetes.io/projected/c4fd1169-a9c3-4c6d-ba92-1a882c7309b0-kube-api-access-p85j8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9zh44\" (UID: \"c4fd1169-a9c3-4c6d-ba92-1a882c7309b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9zh44" Oct 30 18:04:18.991151 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:18.991116 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9zh44" Oct 30 18:04:19.307233 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:19.307206 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9zh44"] Oct 30 18:04:19.309166 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:04:19.309136 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4fd1169_a9c3_4c6d_ba92_1a882c7309b0.slice/crio-3555bcbec1eb131d4ed6002751040b8e7b10be7e9adeae512d196931f89e2163 WatchSource:0}: Error finding container 3555bcbec1eb131d4ed6002751040b8e7b10be7e9adeae512d196931f89e2163: Status 404 returned error can't find the container with id 3555bcbec1eb131d4ed6002751040b8e7b10be7e9adeae512d196931f89e2163 Oct 30 18:04:20.106681 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:20.106649 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9zh44" event={"ID":"c4fd1169-a9c3-4c6d-ba92-1a882c7309b0","Type":"ContainerStarted","Data":"3555bcbec1eb131d4ed6002751040b8e7b10be7e9adeae512d196931f89e2163"} Oct 30 18:04:21.044762 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:21.044667 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3cc458-779ff4f8cf-lt8qr" Oct 30 18:04:21.044958 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:21.044848 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc458-99565d668-p2qbm" Oct 30 18:04:22.484021 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:22.483987 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:04:22.484315 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:22.484052 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:04:23.133230 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:23.133186 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9zh44" event={"ID":"c4fd1169-a9c3-4c6d-ba92-1a882c7309b0","Type":"ContainerStarted","Data":"0319c489f3eda2fd4e8d39c3464e74560110007980be20a8487c1b19b79edb17"} Oct 30 18:04:23.133439 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:23.133291 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9zh44" Oct 30 18:04:23.135725 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:23.135697 4455 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-9zh44 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:04:23.135845 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:23.135754 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9zh44" podUID="c4fd1169-a9c3-4c6d-ba92-1a882c7309b0" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:04:23.216838 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:23.216780 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9zh44" podStartSLOduration=2.045489463 podStartE2EDuration="5.216763599s" totalImagesPullingTime="3.171274136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:04:23.203959448 +0000 UTC m=+715.691596483" watchObservedRunningTime="2026-10-30 18:04:23.216763599 +0000 UTC m=+715.704400632" Oct 30 18:04:24.140389 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:24.140363 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9zh44" Oct 30 18:04:30.385347 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:04:30.385292 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:04:43.322610 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.322565 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j"] Oct 30 18:04:43.327163 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.327139 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" Oct 30 18:04:43.363455 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.363423 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j"] Oct 30 18:04:43.423763 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.423725 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c9b7598-cec3-43ab-90cf-93e7b407d68e-workload-certs\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" Oct 30 18:04:43.423960 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.423771 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vx4g\" (UniqueName: \"kubernetes.io/projected/7c9b7598-cec3-43ab-90cf-93e7b407d68e-kube-api-access-5vx4g\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" Oct 30 18:04:43.423960 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.423920 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c9b7598-cec3-43ab-90cf-93e7b407d68e-istio-podinfo\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" Oct 30 18:04:43.424095 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.424005 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c9b7598-cec3-43ab-90cf-93e7b407d68e-istio-token\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" Oct 30 18:04:43.424095 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.424070 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9b7598-cec3-43ab-90cf-93e7b407d68e-workload-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" Oct 30 18:04:43.424231 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.424204 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c9b7598-cec3-43ab-90cf-93e7b407d68e-istiod-ca-cert\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" Oct 30 18:04:43.424291 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.424242 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c9b7598-cec3-43ab-90cf-93e7b407d68e-istio-envoy\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" Oct 30 18:04:43.424369 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.424312 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c9b7598-cec3-43ab-90cf-93e7b407d68e-istio-data\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" Oct 30 18:04:43.424369 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.424347 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9b7598-cec3-43ab-90cf-93e7b407d68e-credential-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" Oct 30 18:04:43.425024 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.425004 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-6e8fca-openshift-default-dockercfg-sjtwn\"" Oct 30 18:04:43.525481 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.525445 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9b7598-cec3-43ab-90cf-93e7b407d68e-workload-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" Oct 30 18:04:43.525691 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.525521 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c9b7598-cec3-43ab-90cf-93e7b407d68e-istiod-ca-cert\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" Oct 30 18:04:43.525691 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.525551 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c9b7598-cec3-43ab-90cf-93e7b407d68e-istio-envoy\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" Oct 30 18:04:43.525691 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.525575 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c9b7598-cec3-43ab-90cf-93e7b407d68e-istio-data\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" Oct 30 18:04:43.525691 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.525595 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9b7598-cec3-43ab-90cf-93e7b407d68e-credential-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" Oct 30 18:04:43.525691 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.525639 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c9b7598-cec3-43ab-90cf-93e7b407d68e-workload-certs\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" Oct 30 18:04:43.525691 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.525670 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vx4g\" (UniqueName: \"kubernetes.io/projected/7c9b7598-cec3-43ab-90cf-93e7b407d68e-kube-api-access-5vx4g\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" Oct 30 18:04:43.526018 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.525727 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c9b7598-cec3-43ab-90cf-93e7b407d68e-istio-podinfo\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" Oct 30 18:04:43.526018 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.525762 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c9b7598-cec3-43ab-90cf-93e7b407d68e-istio-token\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" Oct 30 18:04:43.526130 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.526025 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9b7598-cec3-43ab-90cf-93e7b407d68e-credential-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" Oct 30 18:04:43.526469 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.526442 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9b7598-cec3-43ab-90cf-93e7b407d68e-workload-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" Oct 30 18:04:43.526610 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.526521 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c9b7598-cec3-43ab-90cf-93e7b407d68e-workload-certs\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" Oct 30 18:04:43.526610 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.526562 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c9b7598-cec3-43ab-90cf-93e7b407d68e-istio-data\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" Oct 30 18:04:43.526716 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.526624 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c9b7598-cec3-43ab-90cf-93e7b407d68e-istiod-ca-cert\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" Oct 30 18:04:43.528243 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.528225 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c9b7598-cec3-43ab-90cf-93e7b407d68e-istio-envoy\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" Oct 30 18:04:43.528746 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.528724 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c9b7598-cec3-43ab-90cf-93e7b407d68e-istio-podinfo\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" Oct 30 18:04:43.549060 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.549028 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c9b7598-cec3-43ab-90cf-93e7b407d68e-istio-token\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" Oct 30 18:04:43.550364 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.550342 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vx4g\" (UniqueName: \"kubernetes.io/projected/7c9b7598-cec3-43ab-90cf-93e7b407d68e-kube-api-access-5vx4g\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" Oct 30 18:04:43.640753 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.640670 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" Oct 30 18:04:43.961433 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.961409 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j"] Oct 30 18:04:43.961834 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:04:43.961811 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c9b7598_cec3_43ab_90cf_93e7b407d68e.slice/crio-828081dbe58ed059499f4ed94440685144eb4355c6f6d16773c6a1d37481b2d6 WatchSource:0}: Error finding container 828081dbe58ed059499f4ed94440685144eb4355c6f6d16773c6a1d37481b2d6: Status 404 returned error can't find the container with id 828081dbe58ed059499f4ed94440685144eb4355c6f6d16773c6a1d37481b2d6 Oct 30 18:04:43.964210 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.964177 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:04:43.964297 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.964241 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:04:43.964297 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:43.964272 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:04:44.275705 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:44.275667 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" event={"ID":"7c9b7598-cec3-43ab-90cf-93e7b407d68e","Type":"ContainerStarted","Data":"524f212977033a1bd2de88b5b493663bfe5755c1ed253ef4a3df7b5be1378a5b"} Oct 30 18:04:44.275916 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:44.275734 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" event={"ID":"7c9b7598-cec3-43ab-90cf-93e7b407d68e","Type":"ContainerStarted","Data":"828081dbe58ed059499f4ed94440685144eb4355c6f6d16773c6a1d37481b2d6"} Oct 30 18:04:44.328834 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:44.328773 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" podStartSLOduration=1.328755663 podStartE2EDuration="1.328755663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:04:44.328123688 +0000 UTC m=+736.815760723" watchObservedRunningTime="2026-10-30 18:04:44.328755663 +0000 UTC m=+736.816392696" Oct 30 18:04:44.640801 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:44.640705 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" Oct 30 18:04:45.642762 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:45.642727 4455 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:04:45.643166 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:45.642794 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" podUID="7c9b7598-cec3-43ab-90cf-93e7b407d68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 18:04:46.641117 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:46.641080 4455 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:04:46.641350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:46.641144 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" podUID="7c9b7598-cec3-43ab-90cf-93e7b407d68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 18:04:47.641615 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:47.641587 4455 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:04:47.642055 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:47.641635 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" podUID="7c9b7598-cec3-43ab-90cf-93e7b407d68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 18:04:48.640975 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:48.640929 4455 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:04:48.641167 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:48.641021 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" podUID="7c9b7598-cec3-43ab-90cf-93e7b407d68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 18:04:49.641124 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:49.641090 4455 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:04:49.641567 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:49.641160 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" podUID="7c9b7598-cec3-43ab-90cf-93e7b407d68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 18:04:50.775344 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:50.774445 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" Oct 30 18:04:50.775858 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:50.775392 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" Oct 30 18:04:50.777378 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:04:50.776722 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" Oct 30 18:05:00.389526 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:05:00.389489 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:05:30.386574 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:05:30.386530 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:05:46.289583 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.289547 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9"] Oct 30 18:05:46.293803 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.293774 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" Oct 30 18:05:46.312084 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.312058 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-419d32-openshift-default-dockercfg-l4pzs\"" Oct 30 18:05:46.329968 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.329934 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9"] Oct 30 18:05:46.365019 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.364972 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c2d808-5274-484e-8520-1f88fa79fc47-workload-certs\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" Oct 30 18:05:46.365216 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.365029 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c2d808-5274-484e-8520-1f88fa79fc47-workload-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" Oct 30 18:05:46.365216 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.365101 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c2d808-5274-484e-8520-1f88fa79fc47-credential-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" Oct 30 18:05:46.365216 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.365154 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c2d808-5274-484e-8520-1f88fa79fc47-istiod-ca-cert\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" Oct 30 18:05:46.365216 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.365177 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwnmj\" (UniqueName: \"kubernetes.io/projected/d2c2d808-5274-484e-8520-1f88fa79fc47-kube-api-access-vwnmj\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" Oct 30 18:05:46.365499 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.365238 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c2d808-5274-484e-8520-1f88fa79fc47-istio-podinfo\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" Oct 30 18:05:46.365499 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.365261 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c2d808-5274-484e-8520-1f88fa79fc47-istio-token\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" Oct 30 18:05:46.365499 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.365382 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c2d808-5274-484e-8520-1f88fa79fc47-istio-envoy\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" Oct 30 18:05:46.365499 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.365399 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c2d808-5274-484e-8520-1f88fa79fc47-istio-data\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" Oct 30 18:05:46.466583 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.466536 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c2d808-5274-484e-8520-1f88fa79fc47-workload-certs\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" Oct 30 18:05:46.466782 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.466633 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c2d808-5274-484e-8520-1f88fa79fc47-workload-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" Oct 30 18:05:46.466782 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.466672 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c2d808-5274-484e-8520-1f88fa79fc47-credential-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" Oct 30 18:05:46.466782 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.466704 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c2d808-5274-484e-8520-1f88fa79fc47-istiod-ca-cert\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" Oct 30 18:05:46.466782 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.466729 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwnmj\" (UniqueName: \"kubernetes.io/projected/d2c2d808-5274-484e-8520-1f88fa79fc47-kube-api-access-vwnmj\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" Oct 30 18:05:46.467016 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.466841 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c2d808-5274-484e-8520-1f88fa79fc47-istio-podinfo\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" Oct 30 18:05:46.467016 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.466888 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c2d808-5274-484e-8520-1f88fa79fc47-istio-token\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" Oct 30 18:05:46.467126 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.467021 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c2d808-5274-484e-8520-1f88fa79fc47-istio-envoy\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" Oct 30 18:05:46.467126 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.467045 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c2d808-5274-484e-8520-1f88fa79fc47-istio-data\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" Oct 30 18:05:46.467126 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.467051 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c2d808-5274-484e-8520-1f88fa79fc47-workload-certs\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" Oct 30 18:05:46.467126 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.467093 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c2d808-5274-484e-8520-1f88fa79fc47-workload-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" Oct 30 18:05:46.467387 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.467124 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c2d808-5274-484e-8520-1f88fa79fc47-credential-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" Oct 30 18:05:46.467530 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.467402 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c2d808-5274-484e-8520-1f88fa79fc47-istio-data\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" Oct 30 18:05:46.467645 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.467621 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c2d808-5274-484e-8520-1f88fa79fc47-istiod-ca-cert\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" Oct 30 18:05:46.469654 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.469625 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c2d808-5274-484e-8520-1f88fa79fc47-istio-envoy\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" Oct 30 18:05:46.469876 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.469856 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c2d808-5274-484e-8520-1f88fa79fc47-istio-podinfo\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" Oct 30 18:05:46.493260 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.493230 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwnmj\" (UniqueName: \"kubernetes.io/projected/d2c2d808-5274-484e-8520-1f88fa79fc47-kube-api-access-vwnmj\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" Oct 30 18:05:46.494872 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.494847 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c2d808-5274-484e-8520-1f88fa79fc47-istio-token\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" Oct 30 18:05:46.608789 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.608708 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" Oct 30 18:05:46.928290 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.928266 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9"] Oct 30 18:05:46.931045 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:05:46.931007 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2c2d808_5274_484e_8520_1f88fa79fc47.slice/crio-ab4e174b785d423cfa8c20dff3c45370d181a767140eeee606e826f9570cda77 WatchSource:0}: Error finding container ab4e174b785d423cfa8c20dff3c45370d181a767140eeee606e826f9570cda77: Status 404 returned error can't find the container with id ab4e174b785d423cfa8c20dff3c45370d181a767140eeee606e826f9570cda77 Oct 30 18:05:46.933834 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.933799 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:05:46.933935 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.933872 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:05:46.933935 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:46.933902 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:05:47.757653 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:47.757618 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" event={"ID":"d2c2d808-5274-484e-8520-1f88fa79fc47","Type":"ContainerStarted","Data":"f5047262e2a0102f0504b4a2a5f81abba68df4165fafedfa87f171beafca6b7c"} Oct 30 18:05:47.758080 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:47.757677 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" event={"ID":"d2c2d808-5274-484e-8520-1f88fa79fc47","Type":"ContainerStarted","Data":"ab4e174b785d423cfa8c20dff3c45370d181a767140eeee606e826f9570cda77"} Oct 30 18:05:48.610465 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:48.610434 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" Oct 30 18:05:48.611008 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:48.610969 4455 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 18:05:48.611216 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:48.611190 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" podUID="d2c2d808-5274-484e-8520-1f88fa79fc47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 18:05:49.610070 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:49.610027 4455 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 18:05:49.610621 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:49.610088 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" podUID="d2c2d808-5274-484e-8520-1f88fa79fc47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 18:05:50.609122 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:50.609087 4455 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 18:05:50.609358 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:50.609158 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" podUID="d2c2d808-5274-484e-8520-1f88fa79fc47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 18:05:51.610032 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:51.609996 4455 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 18:05:51.610525 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:51.610053 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" podUID="d2c2d808-5274-484e-8520-1f88fa79fc47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 18:05:52.609916 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:52.609883 4455 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 18:05:52.610378 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:52.609935 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" podUID="d2c2d808-5274-484e-8520-1f88fa79fc47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 18:05:53.803586 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:53.803551 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" Oct 30 18:05:53.804399 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:53.804374 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" Oct 30 18:05:53.805120 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:53.805100 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" Oct 30 18:05:53.865238 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:05:53.865186 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" podStartSLOduration=7.865172727 podStartE2EDuration="7.865172727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:47.825575393 +0000 UTC m=+800.313212428" watchObservedRunningTime="2026-10-30 18:05:53.865172727 +0000 UTC m=+806.352809809" Oct 30 18:06:00.384990 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:06:00.384949 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:06:04.238289 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.238258 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx"] Oct 30 18:06:04.241827 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.241801 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" Oct 30 18:06:04.266259 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.266232 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b424a5-openshift-default-dockercfg-gfhfl\"" Oct 30 18:06:04.276656 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.276630 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx"] Oct 30 18:06:04.325907 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.325870 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-credential-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" Oct 30 18:06:04.326097 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.325939 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-istio-data\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" Oct 30 18:06:04.326097 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.325975 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-istio-token\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" Oct 30 18:06:04.326097 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.325998 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-workload-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" Oct 30 18:06:04.326097 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.326025 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-istiod-ca-cert\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" Oct 30 18:06:04.326097 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.326084 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-istio-podinfo\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" Oct 30 18:06:04.326578 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.326126 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7rf7\" (UniqueName: \"kubernetes.io/projected/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-kube-api-access-g7rf7\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" Oct 30 18:06:04.326578 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.326270 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-istio-envoy\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" Oct 30 18:06:04.326578 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.326338 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-workload-certs\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" Oct 30 18:06:04.427464 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.427420 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-istio-envoy\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" Oct 30 18:06:04.427464 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.427467 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-workload-certs\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" Oct 30 18:06:04.427722 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.427528 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-credential-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" Oct 30 18:06:04.427722 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.427561 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-istio-data\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" Oct 30 18:06:04.427722 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.427585 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-istio-token\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" Oct 30 18:06:04.427722 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.427606 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-workload-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" Oct 30 18:06:04.427722 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.427633 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-istiod-ca-cert\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" Oct 30 18:06:04.427722 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.427695 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-istio-podinfo\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" Oct 30 18:06:04.427722 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.427723 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7rf7\" (UniqueName: \"kubernetes.io/projected/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-kube-api-access-g7rf7\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" Oct 30 18:06:04.428972 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.428943 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-credential-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" Oct 30 18:06:04.429940 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.428997 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-istio-data\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" Oct 30 18:06:04.429940 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.429189 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-workload-certs\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" Oct 30 18:06:04.430144 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.429422 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-workload-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" Oct 30 18:06:04.430144 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.429870 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-istiod-ca-cert\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" Oct 30 18:06:04.430755 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.430731 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-istio-envoy\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" Oct 30 18:06:04.432445 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.432420 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-istio-podinfo\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" Oct 30 18:06:04.480381 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.480350 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7rf7\" (UniqueName: \"kubernetes.io/projected/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-kube-api-access-g7rf7\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" Oct 30 18:06:04.481608 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.481539 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-istio-token\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" Oct 30 18:06:04.558662 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.558576 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" Oct 30 18:06:04.910436 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.910405 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx"] Oct 30 18:06:04.916851 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.916808 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:06:04.916969 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.916883 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:06:04.916969 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:04.916913 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:06:05.935349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:05.930508 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" event={"ID":"c7c21be6-b9eb-401c-aa3c-af5b62960cd8","Type":"ContainerStarted","Data":"2e57522cd1c5af27c444736562b4275230f8fe90ef6a33f7e7ef157c5cc09714"} Oct 30 18:06:05.935349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:05.930567 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" event={"ID":"c7c21be6-b9eb-401c-aa3c-af5b62960cd8","Type":"ContainerStarted","Data":"23903a47680610f062477ecc91016d0254f146879b0de2dbdab9dd1322580e2b"} Oct 30 18:06:05.986420 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:05.986313 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" podStartSLOduration=1.986293385 podStartE2EDuration="1.986293385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:05.985269638 +0000 UTC m=+818.472906679" watchObservedRunningTime="2026-10-30 18:06:05.986293385 +0000 UTC m=+818.473930424" Oct 30 18:06:06.563167 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:06.563127 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" Oct 30 18:06:06.564910 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:06.564879 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 18:06:06.565051 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:06.564929 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" podUID="c7c21be6-b9eb-401c-aa3c-af5b62960cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 18:06:07.559585 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:07.559546 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 18:06:07.559982 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:07.559607 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" podUID="c7c21be6-b9eb-401c-aa3c-af5b62960cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 18:06:08.559234 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:08.559197 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 18:06:08.559452 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:08.559251 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" podUID="c7c21be6-b9eb-401c-aa3c-af5b62960cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 18:06:09.559123 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:09.559077 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 18:06:09.559740 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:09.559706 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" podUID="c7c21be6-b9eb-401c-aa3c-af5b62960cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 18:06:10.560249 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:10.559734 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 18:06:10.560249 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:10.559802 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" podUID="c7c21be6-b9eb-401c-aa3c-af5b62960cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 18:06:11.560045 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:11.560005 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 18:06:11.560206 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:11.560092 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" podUID="c7c21be6-b9eb-401c-aa3c-af5b62960cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 18:06:12.563943 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:12.563916 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" Oct 30 18:06:12.564427 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:12.564334 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" Oct 30 18:06:12.565014 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:12.564995 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" Oct 30 18:06:24.110902 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.110857 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x"] Oct 30 18:06:24.116772 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.116740 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" Oct 30 18:06:24.131256 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.131230 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-57ee15-openshift-default-dockercfg-jlxgq\"" Oct 30 18:06:24.147633 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.147602 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x"] Oct 30 18:06:24.203341 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.203293 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e972725e-489d-4f2a-a734-27d7eb570858-istio-token\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" Oct 30 18:06:24.203501 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.203377 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6sqd\" (UniqueName: \"kubernetes.io/projected/e972725e-489d-4f2a-a734-27d7eb570858-kube-api-access-r6sqd\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" Oct 30 18:06:24.203501 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.203425 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e972725e-489d-4f2a-a734-27d7eb570858-istio-envoy\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" Oct 30 18:06:24.203501 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.203460 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e972725e-489d-4f2a-a734-27d7eb570858-credential-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" Oct 30 18:06:24.203612 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.203556 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e972725e-489d-4f2a-a734-27d7eb570858-istio-data\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" Oct 30 18:06:24.203655 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.203634 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e972725e-489d-4f2a-a734-27d7eb570858-workload-certs\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" Oct 30 18:06:24.203733 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.203718 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e972725e-489d-4f2a-a734-27d7eb570858-workload-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" Oct 30 18:06:24.203767 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.203751 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e972725e-489d-4f2a-a734-27d7eb570858-istiod-ca-cert\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" Oct 30 18:06:24.203839 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.203825 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e972725e-489d-4f2a-a734-27d7eb570858-istio-podinfo\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" Oct 30 18:06:24.304760 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.304718 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e972725e-489d-4f2a-a734-27d7eb570858-istio-podinfo\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" Oct 30 18:06:24.304969 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.304842 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e972725e-489d-4f2a-a734-27d7eb570858-istio-token\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" Oct 30 18:06:24.304969 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.304881 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6sqd\" (UniqueName: \"kubernetes.io/projected/e972725e-489d-4f2a-a734-27d7eb570858-kube-api-access-r6sqd\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" Oct 30 18:06:24.304969 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.304923 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e972725e-489d-4f2a-a734-27d7eb570858-istio-envoy\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" Oct 30 18:06:24.305151 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.304971 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e972725e-489d-4f2a-a734-27d7eb570858-credential-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" Oct 30 18:06:24.305315 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.305280 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e972725e-489d-4f2a-a734-27d7eb570858-istio-data\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" Oct 30 18:06:24.305485 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.305365 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e972725e-489d-4f2a-a734-27d7eb570858-workload-certs\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" Oct 30 18:06:24.305485 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.305445 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e972725e-489d-4f2a-a734-27d7eb570858-workload-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" Oct 30 18:06:24.305485 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.305474 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e972725e-489d-4f2a-a734-27d7eb570858-istiod-ca-cert\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" Oct 30 18:06:24.305655 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.305553 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e972725e-489d-4f2a-a734-27d7eb570858-credential-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" Oct 30 18:06:24.305655 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.305635 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e972725e-489d-4f2a-a734-27d7eb570858-istio-data\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" Oct 30 18:06:24.305897 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.305869 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e972725e-489d-4f2a-a734-27d7eb570858-workload-certs\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" Oct 30 18:06:24.306275 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.306235 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e972725e-489d-4f2a-a734-27d7eb570858-istiod-ca-cert\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" Oct 30 18:06:24.306520 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.306495 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e972725e-489d-4f2a-a734-27d7eb570858-workload-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" Oct 30 18:06:24.307972 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.307946 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e972725e-489d-4f2a-a734-27d7eb570858-istio-podinfo\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" Oct 30 18:06:24.308077 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.308046 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e972725e-489d-4f2a-a734-27d7eb570858-istio-envoy\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" Oct 30 18:06:24.328285 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.328250 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e972725e-489d-4f2a-a734-27d7eb570858-istio-token\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" Oct 30 18:06:24.328464 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.328304 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6sqd\" (UniqueName: \"kubernetes.io/projected/e972725e-489d-4f2a-a734-27d7eb570858-kube-api-access-r6sqd\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" Oct 30 18:06:24.437781 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.437685 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" Oct 30 18:06:24.803207 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.803181 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x"] Oct 30 18:06:24.804802 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:06:24.804774 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode972725e_489d_4f2a_a734_27d7eb570858.slice/crio-8b87759888df73f134a91e5308819b7deee84f441e13d2d98cbc4b25ea3ecba9 WatchSource:0}: Error finding container 8b87759888df73f134a91e5308819b7deee84f441e13d2d98cbc4b25ea3ecba9: Status 404 returned error can't find the container with id 8b87759888df73f134a91e5308819b7deee84f441e13d2d98cbc4b25ea3ecba9 Oct 30 18:06:24.807403 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.807358 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:06:24.807489 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.807439 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:06:24.807489 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:24.807468 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:06:25.078957 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:25.078856 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" event={"ID":"e972725e-489d-4f2a-a734-27d7eb570858","Type":"ContainerStarted","Data":"946f14782c693823bc55858538440fc8959065207e645894ca6d65a811047082"} Oct 30 18:06:25.078957 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:25.078927 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" event={"ID":"e972725e-489d-4f2a-a734-27d7eb570858","Type":"ContainerStarted","Data":"8b87759888df73f134a91e5308819b7deee84f441e13d2d98cbc4b25ea3ecba9"} Oct 30 18:06:25.123367 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:25.123291 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" podStartSLOduration=1.123277634 podStartE2EDuration="1.123277634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:25.122011042 +0000 UTC m=+837.609648076" watchObservedRunningTime="2026-10-30 18:06:25.123277634 +0000 UTC m=+837.610914668" Oct 30 18:06:25.438353 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:25.438243 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" Oct 30 18:06:25.439700 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:25.439664 4455 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 18:06:25.439819 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:25.439727 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" podUID="e972725e-489d-4f2a-a734-27d7eb570858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 18:06:26.438887 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:26.438850 4455 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 18:06:26.439297 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:26.438928 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" podUID="e972725e-489d-4f2a-a734-27d7eb570858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 18:06:27.438946 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:27.438914 4455 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 18:06:27.439392 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:27.438974 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" podUID="e972725e-489d-4f2a-a734-27d7eb570858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 18:06:28.439003 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:28.438968 4455 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 18:06:28.439409 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:28.439019 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" podUID="e972725e-489d-4f2a-a734-27d7eb570858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 18:06:28.658619 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:28.658587 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98"] Oct 30 18:06:28.663115 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:28.663083 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" Oct 30 18:06:28.723425 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:28.723388 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98"] Oct 30 18:06:28.762297 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:28.762263 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf4bc50-3670-40de-8159-3d8805fdd601-istio-data\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" Oct 30 18:06:28.762297 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:28.762301 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbf4bc50-3670-40de-8159-3d8805fdd601-istio-token\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" Oct 30 18:06:28.762539 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:28.762358 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8w5p\" (UniqueName: \"kubernetes.io/projected/bbf4bc50-3670-40de-8159-3d8805fdd601-kube-api-access-d8w5p\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" Oct 30 18:06:28.762539 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:28.762467 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf4bc50-3670-40de-8159-3d8805fdd601-istio-podinfo\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" Oct 30 18:06:28.762539 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:28.762486 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf4bc50-3670-40de-8159-3d8805fdd601-istiod-ca-cert\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" Oct 30 18:06:28.762539 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:28.762521 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf4bc50-3670-40de-8159-3d8805fdd601-credential-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" Oct 30 18:06:28.762675 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:28.762582 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf4bc50-3670-40de-8159-3d8805fdd601-workload-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" Oct 30 18:06:28.762735 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:28.762707 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf4bc50-3670-40de-8159-3d8805fdd601-workload-certs\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" Oct 30 18:06:28.762849 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:28.762832 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf4bc50-3670-40de-8159-3d8805fdd601-istio-envoy\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" Oct 30 18:06:28.864174 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:28.864139 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf4bc50-3670-40de-8159-3d8805fdd601-istio-envoy\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" Oct 30 18:06:28.864385 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:28.864188 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf4bc50-3670-40de-8159-3d8805fdd601-istio-data\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" Oct 30 18:06:28.864566 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:28.864549 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf4bc50-3670-40de-8159-3d8805fdd601-istio-data\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" Oct 30 18:06:28.864722 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:28.864692 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbf4bc50-3670-40de-8159-3d8805fdd601-istio-token\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" Oct 30 18:06:28.864867 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:28.864770 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8w5p\" (UniqueName: \"kubernetes.io/projected/bbf4bc50-3670-40de-8159-3d8805fdd601-kube-api-access-d8w5p\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" Oct 30 18:06:28.865305 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:28.865283 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf4bc50-3670-40de-8159-3d8805fdd601-istio-podinfo\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" Oct 30 18:06:28.865450 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:28.865356 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf4bc50-3670-40de-8159-3d8805fdd601-istiod-ca-cert\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" Oct 30 18:06:28.865450 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:28.865398 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf4bc50-3670-40de-8159-3d8805fdd601-credential-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" Oct 30 18:06:28.865569 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:28.865449 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf4bc50-3670-40de-8159-3d8805fdd601-workload-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" Oct 30 18:06:28.865620 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:28.865572 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf4bc50-3670-40de-8159-3d8805fdd601-workload-certs\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" Oct 30 18:06:28.877397 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:28.866028 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf4bc50-3670-40de-8159-3d8805fdd601-workload-certs\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" Oct 30 18:06:28.877397 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:28.866141 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf4bc50-3670-40de-8159-3d8805fdd601-credential-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" Oct 30 18:06:28.877397 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:28.866301 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf4bc50-3670-40de-8159-3d8805fdd601-workload-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" Oct 30 18:06:28.877397 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:28.870820 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf4bc50-3670-40de-8159-3d8805fdd601-istio-podinfo\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" Oct 30 18:06:28.877397 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:28.877347 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf4bc50-3670-40de-8159-3d8805fdd601-istiod-ca-cert\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" Oct 30 18:06:28.895117 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:28.895055 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbf4bc50-3670-40de-8159-3d8805fdd601-istio-token\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" Oct 30 18:06:28.896609 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:28.896582 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf4bc50-3670-40de-8159-3d8805fdd601-istio-envoy\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" Oct 30 18:06:28.898033 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:28.898012 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8w5p\" (UniqueName: \"kubernetes.io/projected/bbf4bc50-3670-40de-8159-3d8805fdd601-kube-api-access-d8w5p\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" Oct 30 18:06:29.012092 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:29.012063 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" Oct 30 18:06:29.361727 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:06:29.361682 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf4bc50_3670_40de_8159_3d8805fdd601.slice/crio-e8ecf9856e887677126de03fdf61388c8a4e29d8daad7c96fa5ad30a3d09f977 WatchSource:0}: Error finding container e8ecf9856e887677126de03fdf61388c8a4e29d8daad7c96fa5ad30a3d09f977: Status 404 returned error can't find the container with id e8ecf9856e887677126de03fdf61388c8a4e29d8daad7c96fa5ad30a3d09f977 Oct 30 18:06:29.363595 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:29.363570 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98"] Oct 30 18:06:29.364844 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:29.364813 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:06:29.364929 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:29.364874 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:06:29.364929 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:29.364905 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:06:29.438562 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:29.438529 4455 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 18:06:29.438705 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:29.438601 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" podUID="e972725e-489d-4f2a-a734-27d7eb570858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 18:06:30.140798 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:30.140752 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" event={"ID":"bbf4bc50-3670-40de-8159-3d8805fdd601","Type":"ContainerStarted","Data":"f7d22f95cf4835ebe39cfc8659ce9bf4786d0f2fa16aafda6f92cbf7fcc80cac"} Oct 30 18:06:30.141203 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:30.140825 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" event={"ID":"bbf4bc50-3670-40de-8159-3d8805fdd601","Type":"ContainerStarted","Data":"e8ecf9856e887677126de03fdf61388c8a4e29d8daad7c96fa5ad30a3d09f977"} Oct 30 18:06:30.426835 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:06:30.426737 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:06:30.438287 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:30.438255 4455 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 18:06:30.438514 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:30.438494 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" podUID="e972725e-489d-4f2a-a734-27d7eb570858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 18:06:31.014348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:31.013707 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" Oct 30 18:06:31.029343 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:31.022468 4455 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:06:31.029343 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:31.022537 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" podUID="bbf4bc50-3670-40de-8159-3d8805fdd601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 18:06:31.320676 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:31.320571 4455 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-84.ec2.internal" podUID="da45dfd37aee5c345465266021945ad3" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:06:31.438901 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:31.438865 4455 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 18:06:31.439088 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:31.438935 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" podUID="e972725e-489d-4f2a-a734-27d7eb570858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 18:06:32.013029 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:32.012994 4455 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:06:32.013221 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:32.013049 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" podUID="bbf4bc50-3670-40de-8159-3d8805fdd601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 18:06:32.442982 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:32.442948 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" Oct 30 18:06:32.443453 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:32.443313 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" Oct 30 18:06:32.443926 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:32.443908 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" Oct 30 18:06:32.533381 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:32.533305 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" podStartSLOduration=4.533284132 podStartE2EDuration="4.533284132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:30.217367513 +0000 UTC m=+842.705004548" watchObservedRunningTime="2026-10-30 18:06:32.533284132 +0000 UTC m=+845.020921167" Oct 30 18:06:33.012878 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:33.012836 4455 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:06:33.013089 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:33.012898 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" podUID="bbf4bc50-3670-40de-8159-3d8805fdd601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 18:06:34.013339 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:34.013284 4455 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:06:34.013810 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:34.013357 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" podUID="bbf4bc50-3670-40de-8159-3d8805fdd601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 18:06:35.012739 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:35.012697 4455 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:06:35.012954 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:35.012752 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" podUID="bbf4bc50-3670-40de-8159-3d8805fdd601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 18:06:36.012786 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:36.012752 4455 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:06:36.013291 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:36.012805 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" podUID="bbf4bc50-3670-40de-8159-3d8805fdd601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 18:06:37.018047 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:37.018019 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" Oct 30 18:06:37.018545 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:37.018312 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" Oct 30 18:06:37.019093 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:37.019075 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" Oct 30 18:06:54.391133 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:54.390910 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s"] Oct 30 18:06:54.407805 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:54.407779 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" Oct 30 18:06:54.420807 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:54.420777 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s"] Oct 30 18:06:54.429656 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:54.429624 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-3cc458-openshift-default-dockercfg-ld6nz\"" Oct 30 18:06:54.472732 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:54.472694 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-istiod-ca-cert\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" Oct 30 18:06:54.472931 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:54.472801 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2m4n\" (UniqueName: \"kubernetes.io/projected/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-kube-api-access-g2m4n\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" Oct 30 18:06:54.472931 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:54.472848 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-istio-data\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" Oct 30 18:06:54.472931 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:54.472917 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-workload-certs\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" Oct 30 18:06:54.473102 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:54.472933 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-istio-token\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" Oct 30 18:06:54.473102 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:54.472959 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-istio-envoy\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" Oct 30 18:06:54.473102 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:54.472975 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-workload-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" Oct 30 18:06:54.473256 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:54.473140 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-istio-podinfo\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" Oct 30 18:06:54.473256 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:54.473171 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-credential-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" Oct 30 18:06:54.573836 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:54.573798 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-workload-certs\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" Oct 30 18:06:54.573836 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:54.573850 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-istio-token\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" Oct 30 18:06:54.574093 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:54.573888 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-istio-envoy\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" Oct 30 18:06:54.574093 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:54.573908 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-workload-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" Oct 30 18:06:54.574093 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:54.574007 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-istio-podinfo\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" Oct 30 18:06:54.574093 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:54.574040 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-credential-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" Oct 30 18:06:54.574295 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:54.574124 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-istiod-ca-cert\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" Oct 30 18:06:54.574295 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:54.574178 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2m4n\" (UniqueName: \"kubernetes.io/projected/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-kube-api-access-g2m4n\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" Oct 30 18:06:54.574295 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:54.574217 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-istio-data\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" Oct 30 18:06:54.574994 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:54.574704 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-istio-data\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" Oct 30 18:06:54.574994 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:54.574781 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-credential-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" Oct 30 18:06:54.575197 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:54.575003 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-workload-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" Oct 30 18:06:54.575694 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:54.575637 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-workload-certs\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" Oct 30 18:06:54.575860 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:54.575840 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-istiod-ca-cert\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" Oct 30 18:06:54.577555 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:54.577531 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-istio-podinfo\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" Oct 30 18:06:54.577936 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:54.577917 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-istio-envoy\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" Oct 30 18:06:54.600345 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:54.599765 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-istio-token\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" Oct 30 18:06:54.600484 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:54.600306 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2m4n\" (UniqueName: \"kubernetes.io/projected/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-kube-api-access-g2m4n\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" Oct 30 18:06:54.724858 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:54.724828 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" Oct 30 18:06:55.057582 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:06:55.057540 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e8e077_c9f7_44c8_9dd5_d1c7a2403862.slice/crio-41256cacd0db1c699eb1df7f1c3028fb3f11181a9e69117d289fbb56c6c39f99 WatchSource:0}: Error finding container 41256cacd0db1c699eb1df7f1c3028fb3f11181a9e69117d289fbb56c6c39f99: Status 404 returned error can't find the container with id 41256cacd0db1c699eb1df7f1c3028fb3f11181a9e69117d289fbb56c6c39f99 Oct 30 18:06:55.059021 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:55.058996 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s"] Oct 30 18:06:55.060098 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:55.060066 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:06:55.060163 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:55.060135 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:06:55.060199 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:55.060166 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:06:55.380439 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:55.380351 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" event={"ID":"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862","Type":"ContainerStarted","Data":"909682ad7c983b900fe1e1f104fd35ebac75d470a756e2922f03fdbae688d41f"} Oct 30 18:06:55.380439 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:55.380400 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" event={"ID":"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862","Type":"ContainerStarted","Data":"41256cacd0db1c699eb1df7f1c3028fb3f11181a9e69117d289fbb56c6c39f99"} Oct 30 18:06:55.435512 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:55.435448 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" podStartSLOduration=1.43542856 podStartE2EDuration="1.43542856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:55.430765406 +0000 UTC m=+867.918402440" watchObservedRunningTime="2026-10-30 18:06:55.43542856 +0000 UTC m=+867.923065595" Oct 30 18:06:55.725010 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:55.724979 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" Oct 30 18:06:56.727685 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:56.727647 4455 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 18:06:56.728170 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:56.727701 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" podUID="d4e8e077-c9f7-44c8-9dd5-d1c7a2403862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 18:06:57.726241 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:57.726209 4455 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 18:06:57.726469 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:57.726280 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" podUID="d4e8e077-c9f7-44c8-9dd5-d1c7a2403862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 18:06:58.726803 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:58.726758 4455 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 18:06:58.727338 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:58.726847 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" podUID="d4e8e077-c9f7-44c8-9dd5-d1c7a2403862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 18:06:59.725833 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:59.725801 4455 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 18:06:59.726040 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:06:59.725864 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" podUID="d4e8e077-c9f7-44c8-9dd5-d1c7a2403862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 18:07:00.409794 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:07:00.409750 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:07:00.725669 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:00.725632 4455 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 18:07:00.725865 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:00.725717 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" podUID="d4e8e077-c9f7-44c8-9dd5-d1c7a2403862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 18:07:02.048849 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:02.048818 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" Oct 30 18:07:02.049673 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:02.049648 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" Oct 30 18:07:02.050386 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:02.050365 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" Oct 30 18:07:04.435033 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:04.434995 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx"] Oct 30 18:07:04.440722 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:04.440700 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" Oct 30 18:07:04.476005 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:04.475970 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx"] Oct 30 18:07:04.536592 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:04.536551 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7b0c73-0041-4549-b47e-76a7aaed2b56-istio-envoy\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" Oct 30 18:07:04.536774 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:04.536613 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7b0c73-0041-4549-b47e-76a7aaed2b56-istio-podinfo\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" Oct 30 18:07:04.536774 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:04.536685 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v685\" (UniqueName: \"kubernetes.io/projected/4e7b0c73-0041-4549-b47e-76a7aaed2b56-kube-api-access-9v685\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" Oct 30 18:07:04.536849 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:04.536803 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7b0c73-0041-4549-b47e-76a7aaed2b56-istio-data\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" Oct 30 18:07:04.536888 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:04.536856 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e7b0c73-0041-4549-b47e-76a7aaed2b56-istio-token\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" Oct 30 18:07:04.536888 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:04.536881 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7b0c73-0041-4549-b47e-76a7aaed2b56-istiod-ca-cert\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" Oct 30 18:07:04.536955 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:04.536904 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7b0c73-0041-4549-b47e-76a7aaed2b56-credential-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" Oct 30 18:07:04.537049 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:04.537037 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7b0c73-0041-4549-b47e-76a7aaed2b56-workload-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" Oct 30 18:07:04.537087 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:04.537066 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7b0c73-0041-4549-b47e-76a7aaed2b56-workload-certs\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" Oct 30 18:07:04.638234 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:04.638201 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7b0c73-0041-4549-b47e-76a7aaed2b56-credential-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" Oct 30 18:07:04.638425 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:04.638266 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7b0c73-0041-4549-b47e-76a7aaed2b56-workload-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" Oct 30 18:07:04.638425 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:04.638299 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7b0c73-0041-4549-b47e-76a7aaed2b56-workload-certs\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" Oct 30 18:07:04.638425 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:04.638381 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7b0c73-0041-4549-b47e-76a7aaed2b56-istio-envoy\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" Oct 30 18:07:04.638573 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:04.638444 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7b0c73-0041-4549-b47e-76a7aaed2b56-istio-podinfo\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" Oct 30 18:07:04.638573 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:04.638466 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v685\" (UniqueName: \"kubernetes.io/projected/4e7b0c73-0041-4549-b47e-76a7aaed2b56-kube-api-access-9v685\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" Oct 30 18:07:04.638670 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:04.638630 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7b0c73-0041-4549-b47e-76a7aaed2b56-credential-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" Oct 30 18:07:04.638726 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:04.638690 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7b0c73-0041-4549-b47e-76a7aaed2b56-workload-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" Oct 30 18:07:04.638785 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:04.638756 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7b0c73-0041-4549-b47e-76a7aaed2b56-istio-data\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" Oct 30 18:07:04.638785 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:04.638768 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7b0c73-0041-4549-b47e-76a7aaed2b56-workload-certs\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" Oct 30 18:07:04.638891 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:04.638856 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e7b0c73-0041-4549-b47e-76a7aaed2b56-istio-token\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" Oct 30 18:07:04.638942 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:04.638917 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7b0c73-0041-4549-b47e-76a7aaed2b56-istiod-ca-cert\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" Oct 30 18:07:04.639129 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:04.639106 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7b0c73-0041-4549-b47e-76a7aaed2b56-istio-data\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" Oct 30 18:07:04.639570 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:04.639551 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7b0c73-0041-4549-b47e-76a7aaed2b56-istiod-ca-cert\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" Oct 30 18:07:04.641389 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:04.641366 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7b0c73-0041-4549-b47e-76a7aaed2b56-istio-envoy\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" Oct 30 18:07:04.641630 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:04.641613 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7b0c73-0041-4549-b47e-76a7aaed2b56-istio-podinfo\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" Oct 30 18:07:04.663554 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:04.663527 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e7b0c73-0041-4549-b47e-76a7aaed2b56-istio-token\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" Oct 30 18:07:04.663823 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:04.663799 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v685\" (UniqueName: \"kubernetes.io/projected/4e7b0c73-0041-4549-b47e-76a7aaed2b56-kube-api-access-9v685\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" Oct 30 18:07:04.757101 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:04.757064 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" Oct 30 18:07:05.117219 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:07:05.117187 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e7b0c73_0041_4549_b47e_76a7aaed2b56.slice/crio-e90f0a27cd278d5d9dc55ef968521a7d31572d1883533571b84f071703d19156 WatchSource:0}: Error finding container e90f0a27cd278d5d9dc55ef968521a7d31572d1883533571b84f071703d19156: Status 404 returned error can't find the container with id e90f0a27cd278d5d9dc55ef968521a7d31572d1883533571b84f071703d19156 Oct 30 18:07:05.119820 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:05.119786 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:07:05.119932 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:05.119872 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:07:05.119932 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:05.119920 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:07:05.120044 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:05.120027 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx"] Oct 30 18:07:05.462374 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:05.462266 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" event={"ID":"4e7b0c73-0041-4549-b47e-76a7aaed2b56","Type":"ContainerStarted","Data":"79d17f879566861647cf69447a10eeff41d3d2ff0ccd53c84938d1107672ab0f"} Oct 30 18:07:05.462374 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:05.462357 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" event={"ID":"4e7b0c73-0041-4549-b47e-76a7aaed2b56","Type":"ContainerStarted","Data":"e90f0a27cd278d5d9dc55ef968521a7d31572d1883533571b84f071703d19156"} Oct 30 18:07:05.514157 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:05.514109 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" podStartSLOduration=1.51409373 podStartE2EDuration="1.51409373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:05.51191526 +0000 UTC m=+877.999552297" watchObservedRunningTime="2026-10-30 18:07:05.51409373 +0000 UTC m=+878.001730763" Oct 30 18:07:05.757749 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:05.757720 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" Oct 30 18:07:06.759537 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:06.759499 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 18:07:06.760009 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:06.759574 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" podUID="4e7b0c73-0041-4549-b47e-76a7aaed2b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 18:07:07.757700 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:07.757664 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 18:07:07.757908 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:07.757726 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" podUID="4e7b0c73-0041-4549-b47e-76a7aaed2b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 18:07:08.758157 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:08.758117 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 18:07:08.758593 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:08.758184 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" podUID="4e7b0c73-0041-4549-b47e-76a7aaed2b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 18:07:09.813148 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:09.812765 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 18:07:09.813148 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:09.812829 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" podUID="4e7b0c73-0041-4549-b47e-76a7aaed2b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 18:07:10.758730 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:10.758689 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 18:07:10.759075 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:10.759034 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" podUID="4e7b0c73-0041-4549-b47e-76a7aaed2b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 18:07:11.762894 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:11.762852 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 18:07:11.763577 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:11.763543 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" podUID="4e7b0c73-0041-4549-b47e-76a7aaed2b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 18:07:12.941261 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:12.941232 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" Oct 30 18:07:12.941725 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:12.941650 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" Oct 30 18:07:12.942531 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:12.942510 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" Oct 30 18:07:22.270044 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:22.269994 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-54fff678bb-bfh89"] Oct 30 18:07:22.275482 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:22.275456 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-54fff678bb-bfh89" Oct 30 18:07:22.299495 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:22.299461 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-54fff678bb-bfh89"] Oct 30 18:07:22.306646 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:22.306616 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 18:07:22.306814 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:22.306684 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-24576edb8e27\"" Oct 30 18:07:22.306814 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:22.306754 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 18:07:22.306814 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:22.306803 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 18:07:22.306993 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:22.306856 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 18:07:22.308694 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:22.308673 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-hzm64\"" Oct 30 18:07:22.397120 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:22.397084 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2fbcb62-dc0f-441a-810c-bb742f7da339-provider-credentials\") pod \"payload-processing-external-model-54fff678bb-bfh89\" (UID: \"d2fbcb62-dc0f-441a-810c-bb742f7da339\") " pod="models-as-a-service/payload-processing-external-model-54fff678bb-bfh89" Oct 30 18:07:22.397313 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:22.397161 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjbmj\" (UniqueName: \"kubernetes.io/projected/d2fbcb62-dc0f-441a-810c-bb742f7da339-kube-api-access-kjbmj\") pod \"payload-processing-external-model-54fff678bb-bfh89\" (UID: \"d2fbcb62-dc0f-441a-810c-bb742f7da339\") " pod="models-as-a-service/payload-processing-external-model-54fff678bb-bfh89" Oct 30 18:07:22.397313 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:22.397270 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2fbcb62-dc0f-441a-810c-bb742f7da339-routing-overlay-volume\") pod \"payload-processing-external-model-54fff678bb-bfh89\" (UID: \"d2fbcb62-dc0f-441a-810c-bb742f7da339\") " pod="models-as-a-service/payload-processing-external-model-54fff678bb-bfh89" Oct 30 18:07:22.397468 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:22.397451 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2fbcb62-dc0f-441a-810c-bb742f7da339-plugins-config-volume\") pod \"payload-processing-external-model-54fff678bb-bfh89\" (UID: \"d2fbcb62-dc0f-441a-810c-bb742f7da339\") " pod="models-as-a-service/payload-processing-external-model-54fff678bb-bfh89" Oct 30 18:07:22.506345 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:22.499287 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2fbcb62-dc0f-441a-810c-bb742f7da339-plugins-config-volume\") pod \"payload-processing-external-model-54fff678bb-bfh89\" (UID: \"d2fbcb62-dc0f-441a-810c-bb742f7da339\") " pod="models-as-a-service/payload-processing-external-model-54fff678bb-bfh89" Oct 30 18:07:22.506345 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:22.499364 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2fbcb62-dc0f-441a-810c-bb742f7da339-provider-credentials\") pod \"payload-processing-external-model-54fff678bb-bfh89\" (UID: \"d2fbcb62-dc0f-441a-810c-bb742f7da339\") " pod="models-as-a-service/payload-processing-external-model-54fff678bb-bfh89" Oct 30 18:07:22.506345 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:22.499443 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjbmj\" (UniqueName: \"kubernetes.io/projected/d2fbcb62-dc0f-441a-810c-bb742f7da339-kube-api-access-kjbmj\") pod \"payload-processing-external-model-54fff678bb-bfh89\" (UID: \"d2fbcb62-dc0f-441a-810c-bb742f7da339\") " pod="models-as-a-service/payload-processing-external-model-54fff678bb-bfh89" Oct 30 18:07:22.506345 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:22.499576 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2fbcb62-dc0f-441a-810c-bb742f7da339-routing-overlay-volume\") pod \"payload-processing-external-model-54fff678bb-bfh89\" (UID: \"d2fbcb62-dc0f-441a-810c-bb742f7da339\") " pod="models-as-a-service/payload-processing-external-model-54fff678bb-bfh89" Oct 30 18:07:22.506345 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:22.500643 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2fbcb62-dc0f-441a-810c-bb742f7da339-routing-overlay-volume\") pod \"payload-processing-external-model-54fff678bb-bfh89\" (UID: \"d2fbcb62-dc0f-441a-810c-bb742f7da339\") " pod="models-as-a-service/payload-processing-external-model-54fff678bb-bfh89" Oct 30 18:07:22.506345 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:22.501384 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2fbcb62-dc0f-441a-810c-bb742f7da339-plugins-config-volume\") pod \"payload-processing-external-model-54fff678bb-bfh89\" (UID: \"d2fbcb62-dc0f-441a-810c-bb742f7da339\") " pod="models-as-a-service/payload-processing-external-model-54fff678bb-bfh89" Oct 30 18:07:22.506957 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:22.506663 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2fbcb62-dc0f-441a-810c-bb742f7da339-provider-credentials\") pod \"payload-processing-external-model-54fff678bb-bfh89\" (UID: \"d2fbcb62-dc0f-441a-810c-bb742f7da339\") " pod="models-as-a-service/payload-processing-external-model-54fff678bb-bfh89" Oct 30 18:07:22.551154 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:22.551077 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjbmj\" (UniqueName: \"kubernetes.io/projected/d2fbcb62-dc0f-441a-810c-bb742f7da339-kube-api-access-kjbmj\") pod \"payload-processing-external-model-54fff678bb-bfh89\" (UID: \"d2fbcb62-dc0f-441a-810c-bb742f7da339\") " pod="models-as-a-service/payload-processing-external-model-54fff678bb-bfh89" Oct 30 18:07:22.598593 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:22.598551 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-54fff678bb-bfh89" Oct 30 18:07:22.930543 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:07:22.930512 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2fbcb62_dc0f_441a_810c_bb742f7da339.slice/crio-1ea9ed387f35fda284033d73aa654bc6001715d55d8429f2136584eee1dbe9a8 WatchSource:0}: Error finding container 1ea9ed387f35fda284033d73aa654bc6001715d55d8429f2136584eee1dbe9a8: Status 404 returned error can't find the container with id 1ea9ed387f35fda284033d73aa654bc6001715d55d8429f2136584eee1dbe9a8 Oct 30 18:07:22.931074 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:22.931055 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-54fff678bb-bfh89"] Oct 30 18:07:23.663063 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:23.663030 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-54fff678bb-bfh89" event={"ID":"d2fbcb62-dc0f-441a-810c-bb742f7da339","Type":"ContainerStarted","Data":"170f670342e7a7fabccb67eee03fbb5a37f1cec607b62804dc2525f9d326f9a0"} Oct 30 18:07:23.663501 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:23.663094 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-54fff678bb-bfh89" event={"ID":"d2fbcb62-dc0f-441a-810c-bb742f7da339","Type":"ContainerStarted","Data":"1ea9ed387f35fda284033d73aa654bc6001715d55d8429f2136584eee1dbe9a8"} Oct 30 18:07:23.663501 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:23.663233 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-54fff678bb-bfh89" Oct 30 18:07:23.708672 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:23.708620 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-54fff678bb-bfh89" podStartSLOduration=1.70860108 podStartE2EDuration="1.70860108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:23.708419135 +0000 UTC m=+896.196056170" watchObservedRunningTime="2026-10-30 18:07:23.70860108 +0000 UTC m=+896.196238114" Oct 30 18:07:28.900481 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:28.900442 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf"] Oct 30 18:07:28.906063 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:28.906042 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" Oct 30 18:07:28.953551 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:28.953520 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf"] Oct 30 18:07:29.020883 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:29.020841 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zndlf\" (UniqueName: \"kubernetes.io/projected/3daa8589-9def-486f-82ee-0c923a18b68b-kube-api-access-zndlf\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" Oct 30 18:07:29.021103 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:29.020893 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3daa8589-9def-486f-82ee-0c923a18b68b-workload-certs\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" Oct 30 18:07:29.021103 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:29.021064 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daa8589-9def-486f-82ee-0c923a18b68b-credential-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" Oct 30 18:07:29.021588 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:29.021110 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daa8589-9def-486f-82ee-0c923a18b68b-workload-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" Oct 30 18:07:29.021588 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:29.021144 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3daa8589-9def-486f-82ee-0c923a18b68b-istio-data\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" Oct 30 18:07:29.021588 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:29.021207 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3daa8589-9def-486f-82ee-0c923a18b68b-istio-token\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" Oct 30 18:07:29.021588 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:29.021307 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3daa8589-9def-486f-82ee-0c923a18b68b-istio-podinfo\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" Oct 30 18:07:29.021588 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:29.021353 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3daa8589-9def-486f-82ee-0c923a18b68b-istiod-ca-cert\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" Oct 30 18:07:29.021588 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:29.021472 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3daa8589-9def-486f-82ee-0c923a18b68b-istio-envoy\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" Oct 30 18:07:29.122360 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:29.122289 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daa8589-9def-486f-82ee-0c923a18b68b-credential-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" Oct 30 18:07:29.122573 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:29.122377 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daa8589-9def-486f-82ee-0c923a18b68b-workload-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" Oct 30 18:07:29.122573 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:29.122400 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3daa8589-9def-486f-82ee-0c923a18b68b-istio-data\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" Oct 30 18:07:29.122573 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:29.122452 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3daa8589-9def-486f-82ee-0c923a18b68b-istio-token\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" Oct 30 18:07:29.122573 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:29.122554 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3daa8589-9def-486f-82ee-0c923a18b68b-istio-podinfo\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" Oct 30 18:07:29.122791 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:29.122574 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3daa8589-9def-486f-82ee-0c923a18b68b-istiod-ca-cert\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" Oct 30 18:07:29.122791 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:29.122684 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3daa8589-9def-486f-82ee-0c923a18b68b-istio-envoy\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" Oct 30 18:07:29.122791 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:29.122745 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zndlf\" (UniqueName: \"kubernetes.io/projected/3daa8589-9def-486f-82ee-0c923a18b68b-kube-api-access-zndlf\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" Oct 30 18:07:29.122791 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:29.122755 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daa8589-9def-486f-82ee-0c923a18b68b-credential-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" Oct 30 18:07:29.122791 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:29.122772 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3daa8589-9def-486f-82ee-0c923a18b68b-workload-certs\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" Oct 30 18:07:29.122791 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:29.122771 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3daa8589-9def-486f-82ee-0c923a18b68b-istio-data\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" Oct 30 18:07:29.123123 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:29.122838 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daa8589-9def-486f-82ee-0c923a18b68b-workload-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" Oct 30 18:07:29.123123 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:29.122997 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3daa8589-9def-486f-82ee-0c923a18b68b-workload-certs\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" Oct 30 18:07:29.123451 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:29.123430 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3daa8589-9def-486f-82ee-0c923a18b68b-istiod-ca-cert\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" Oct 30 18:07:29.125067 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:29.125043 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3daa8589-9def-486f-82ee-0c923a18b68b-istio-envoy\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" Oct 30 18:07:29.125580 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:29.125557 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3daa8589-9def-486f-82ee-0c923a18b68b-istio-podinfo\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" Oct 30 18:07:29.147864 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:29.147831 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3daa8589-9def-486f-82ee-0c923a18b68b-istio-token\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" Oct 30 18:07:29.147986 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:29.147967 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zndlf\" (UniqueName: \"kubernetes.io/projected/3daa8589-9def-486f-82ee-0c923a18b68b-kube-api-access-zndlf\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" Oct 30 18:07:29.220905 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:29.220873 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" Oct 30 18:07:29.544600 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:07:29.544558 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3daa8589_9def_486f_82ee_0c923a18b68b.slice/crio-ad191e98f17f00ba8e5a4ab55853ec4faf59e0449ca435b8fb612d0c1394a7d8 WatchSource:0}: Error finding container ad191e98f17f00ba8e5a4ab55853ec4faf59e0449ca435b8fb612d0c1394a7d8: Status 404 returned error can't find the container with id ad191e98f17f00ba8e5a4ab55853ec4faf59e0449ca435b8fb612d0c1394a7d8 Oct 30 18:07:29.547109 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:29.547066 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:07:29.547190 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:29.547145 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:07:29.547190 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:29.547178 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:07:29.547533 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:29.547506 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf"] Oct 30 18:07:29.714841 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:29.714798 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" event={"ID":"3daa8589-9def-486f-82ee-0c923a18b68b","Type":"ContainerStarted","Data":"9736a79721eca5b603182cff32bfc216e3c642b536bb8178a0f7d54b8017d80f"} Oct 30 18:07:29.715038 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:29.714866 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" event={"ID":"3daa8589-9def-486f-82ee-0c923a18b68b","Type":"ContainerStarted","Data":"ad191e98f17f00ba8e5a4ab55853ec4faf59e0449ca435b8fb612d0c1394a7d8"} Oct 30 18:07:29.764474 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:29.764423 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" podStartSLOduration=1.764409649 podStartE2EDuration="1.764409649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:29.763829204 +0000 UTC m=+902.251466237" watchObservedRunningTime="2026-10-30 18:07:29.764409649 +0000 UTC m=+902.252046705" Oct 30 18:07:30.221438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:30.221402 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" Oct 30 18:07:30.223066 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:30.223040 4455 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 18:07:30.223208 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:30.223131 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" podUID="3daa8589-9def-486f-82ee-0c923a18b68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 18:07:30.417033 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:07:30.417000 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:07:31.222216 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:31.222182 4455 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 18:07:31.222640 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:31.222240 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" podUID="3daa8589-9def-486f-82ee-0c923a18b68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 18:07:31.603065 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:31.602969 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw"] Oct 30 18:07:31.607440 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:31.607417 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" Oct 30 18:07:31.639931 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:31.639904 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw"] Oct 30 18:07:31.655585 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:31.655551 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ee94ddb-0834-4ee6-835c-4363b672e1c2-workload-certs\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" Oct 30 18:07:31.655905 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:31.655885 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ee94ddb-0834-4ee6-835c-4363b672e1c2-istio-token\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" Oct 30 18:07:31.656085 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:31.656070 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ee94ddb-0834-4ee6-835c-4363b672e1c2-istio-podinfo\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" Oct 30 18:07:31.656222 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:31.656208 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ee94ddb-0834-4ee6-835c-4363b672e1c2-istiod-ca-cert\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" Oct 30 18:07:31.656710 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:31.656691 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee94ddb-0834-4ee6-835c-4363b672e1c2-credential-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" Oct 30 18:07:31.656863 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:31.656847 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ee94ddb-0834-4ee6-835c-4363b672e1c2-istio-envoy\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" Oct 30 18:07:31.657008 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:31.656979 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ee94ddb-0834-4ee6-835c-4363b672e1c2-istio-data\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" Oct 30 18:07:31.657158 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:31.657143 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee94ddb-0834-4ee6-835c-4363b672e1c2-workload-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" Oct 30 18:07:31.657349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:31.657295 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb8g6\" (UniqueName: \"kubernetes.io/projected/1ee94ddb-0834-4ee6-835c-4363b672e1c2-kube-api-access-wb8g6\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" Oct 30 18:07:31.758685 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:31.758645 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb8g6\" (UniqueName: \"kubernetes.io/projected/1ee94ddb-0834-4ee6-835c-4363b672e1c2-kube-api-access-wb8g6\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" Oct 30 18:07:31.758880 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:31.758770 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ee94ddb-0834-4ee6-835c-4363b672e1c2-workload-certs\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" Oct 30 18:07:31.758880 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:31.758831 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ee94ddb-0834-4ee6-835c-4363b672e1c2-istio-token\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" Oct 30 18:07:31.758880 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:31.758857 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ee94ddb-0834-4ee6-835c-4363b672e1c2-istio-podinfo\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" Oct 30 18:07:31.758880 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:31.758874 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ee94ddb-0834-4ee6-835c-4363b672e1c2-istiod-ca-cert\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" Oct 30 18:07:31.759113 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:31.758893 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee94ddb-0834-4ee6-835c-4363b672e1c2-credential-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" Oct 30 18:07:31.759113 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:31.758920 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ee94ddb-0834-4ee6-835c-4363b672e1c2-istio-envoy\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" Oct 30 18:07:31.759113 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:31.758945 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ee94ddb-0834-4ee6-835c-4363b672e1c2-istio-data\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" Oct 30 18:07:31.759113 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:31.759031 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee94ddb-0834-4ee6-835c-4363b672e1c2-workload-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" Oct 30 18:07:31.759497 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:31.759467 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ee94ddb-0834-4ee6-835c-4363b672e1c2-workload-certs\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" Oct 30 18:07:31.759667 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:31.759546 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ee94ddb-0834-4ee6-835c-4363b672e1c2-istio-data\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" Oct 30 18:07:31.759958 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:31.759820 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee94ddb-0834-4ee6-835c-4363b672e1c2-workload-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" Oct 30 18:07:31.760096 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:31.759903 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ee94ddb-0834-4ee6-835c-4363b672e1c2-istiod-ca-cert\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" Oct 30 18:07:31.760213 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:31.759950 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee94ddb-0834-4ee6-835c-4363b672e1c2-credential-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" Oct 30 18:07:31.761535 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:31.761513 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ee94ddb-0834-4ee6-835c-4363b672e1c2-istio-envoy\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" Oct 30 18:07:31.762029 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:31.762008 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ee94ddb-0834-4ee6-835c-4363b672e1c2-istio-podinfo\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" Oct 30 18:07:31.800348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:31.797884 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb8g6\" (UniqueName: \"kubernetes.io/projected/1ee94ddb-0834-4ee6-835c-4363b672e1c2-kube-api-access-wb8g6\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" Oct 30 18:07:31.805855 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:31.805792 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ee94ddb-0834-4ee6-835c-4363b672e1c2-istio-token\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" Oct 30 18:07:31.922498 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:31.922409 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" Oct 30 18:07:32.222338 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:32.222285 4455 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 18:07:32.222807 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:32.222362 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" podUID="3daa8589-9def-486f-82ee-0c923a18b68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 18:07:32.265702 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:07:32.265664 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ee94ddb_0834_4ee6_835c_4363b672e1c2.slice/crio-4fecb219a18959e281b70c8eb6b506851cd427941fdb53df29dd20619ce55d35 WatchSource:0}: Error finding container 4fecb219a18959e281b70c8eb6b506851cd427941fdb53df29dd20619ce55d35: Status 404 returned error can't find the container with id 4fecb219a18959e281b70c8eb6b506851cd427941fdb53df29dd20619ce55d35 Oct 30 18:07:32.274218 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:32.273034 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw"] Oct 30 18:07:32.276597 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:32.275284 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:07:32.276597 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:32.275395 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:07:32.276597 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:32.275441 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:07:32.743121 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:32.743061 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" event={"ID":"1ee94ddb-0834-4ee6-835c-4363b672e1c2","Type":"ContainerStarted","Data":"d2756a38662fd459b3cc2b877e9d404d450bcb728d8be9cd1573cb33983920d5"} Oct 30 18:07:32.743296 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:32.743141 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" event={"ID":"1ee94ddb-0834-4ee6-835c-4363b672e1c2","Type":"ContainerStarted","Data":"4fecb219a18959e281b70c8eb6b506851cd427941fdb53df29dd20619ce55d35"} Oct 30 18:07:32.805171 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:32.805094 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" podStartSLOduration=1.805070659 podStartE2EDuration="1.805070659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:32.793929309 +0000 UTC m=+905.281566342" watchObservedRunningTime="2026-10-30 18:07:32.805070659 +0000 UTC m=+905.292707697" Oct 30 18:07:32.922716 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:32.922668 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" Oct 30 18:07:33.223381 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:33.222777 4455 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 18:07:33.223381 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:33.222840 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" podUID="3daa8589-9def-486f-82ee-0c923a18b68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 18:07:33.924366 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:33.924312 4455 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 18:07:33.924568 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:33.924386 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" podUID="1ee94ddb-0834-4ee6-835c-4363b672e1c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 18:07:34.222818 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:34.222780 4455 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 18:07:34.223128 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:34.223094 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" podUID="3daa8589-9def-486f-82ee-0c923a18b68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 18:07:34.703035 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:34.701814 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-54fff678bb-bfh89" Oct 30 18:07:34.936368 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:34.934983 4455 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 18:07:34.936368 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:34.935045 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" podUID="1ee94ddb-0834-4ee6-835c-4363b672e1c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 18:07:35.230885 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:35.230844 4455 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 18:07:35.231191 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:35.231161 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" podUID="3daa8589-9def-486f-82ee-0c923a18b68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 18:07:35.923680 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:35.923648 4455 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 18:07:35.924048 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:35.923705 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" podUID="1ee94ddb-0834-4ee6-835c-4363b672e1c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 18:07:36.222542 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:36.222507 4455 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 18:07:36.222817 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:36.222785 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" podUID="3daa8589-9def-486f-82ee-0c923a18b68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 18:07:36.923521 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:36.923489 4455 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 18:07:36.923722 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:36.923547 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" podUID="1ee94ddb-0834-4ee6-835c-4363b672e1c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 18:07:37.225928 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:37.225898 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" Oct 30 18:07:37.226361 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:37.226344 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" Oct 30 18:07:37.227034 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:37.227012 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" Oct 30 18:07:37.922952 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:37.922911 4455 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 18:07:37.923158 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:37.922983 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" podUID="1ee94ddb-0834-4ee6-835c-4363b672e1c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 18:07:38.930757 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:38.930722 4455 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 18:07:38.931353 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:38.931297 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" podUID="1ee94ddb-0834-4ee6-835c-4363b672e1c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 18:07:39.414351 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:39.412274 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6c5884dffd-mcl69"] Oct 30 18:07:39.436101 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:39.436059 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c5884dffd-mcl69" Oct 30 18:07:39.457959 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:39.457924 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-25c3ad6fd807\"" Oct 30 18:07:39.502212 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:39.502162 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6893ebba-1b72-4c8e-a877-13904ceb0bfc-routing-overlay-volume\") pod \"payload-processing-external-model-6c5884dffd-mcl69\" (UID: \"6893ebba-1b72-4c8e-a877-13904ceb0bfc\") " pod="models-as-a-service/payload-processing-external-model-6c5884dffd-mcl69" Oct 30 18:07:39.513421 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:39.513384 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsflp\" (UniqueName: \"kubernetes.io/projected/6893ebba-1b72-4c8e-a877-13904ceb0bfc-kube-api-access-wsflp\") pod \"payload-processing-external-model-6c5884dffd-mcl69\" (UID: \"6893ebba-1b72-4c8e-a877-13904ceb0bfc\") " pod="models-as-a-service/payload-processing-external-model-6c5884dffd-mcl69" Oct 30 18:07:39.513756 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:39.513731 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6893ebba-1b72-4c8e-a877-13904ceb0bfc-provider-credentials\") pod \"payload-processing-external-model-6c5884dffd-mcl69\" (UID: \"6893ebba-1b72-4c8e-a877-13904ceb0bfc\") " pod="models-as-a-service/payload-processing-external-model-6c5884dffd-mcl69" Oct 30 18:07:39.514091 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:39.514065 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6893ebba-1b72-4c8e-a877-13904ceb0bfc-plugins-config-volume\") pod \"payload-processing-external-model-6c5884dffd-mcl69\" (UID: \"6893ebba-1b72-4c8e-a877-13904ceb0bfc\") " pod="models-as-a-service/payload-processing-external-model-6c5884dffd-mcl69" Oct 30 18:07:39.519773 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:39.519687 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c5884dffd-mcl69"] Oct 30 18:07:39.752805 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:39.752767 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6893ebba-1b72-4c8e-a877-13904ceb0bfc-plugins-config-volume\") pod \"payload-processing-external-model-6c5884dffd-mcl69\" (UID: \"6893ebba-1b72-4c8e-a877-13904ceb0bfc\") " pod="models-as-a-service/payload-processing-external-model-6c5884dffd-mcl69" Oct 30 18:07:39.757282 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:39.757249 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6893ebba-1b72-4c8e-a877-13904ceb0bfc-plugins-config-volume\") pod \"payload-processing-external-model-6c5884dffd-mcl69\" (UID: \"6893ebba-1b72-4c8e-a877-13904ceb0bfc\") " pod="models-as-a-service/payload-processing-external-model-6c5884dffd-mcl69" Oct 30 18:07:39.772480 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:39.772445 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6893ebba-1b72-4c8e-a877-13904ceb0bfc-routing-overlay-volume\") pod \"payload-processing-external-model-6c5884dffd-mcl69\" (UID: \"6893ebba-1b72-4c8e-a877-13904ceb0bfc\") " pod="models-as-a-service/payload-processing-external-model-6c5884dffd-mcl69" Oct 30 18:07:39.772791 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:39.772768 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsflp\" (UniqueName: \"kubernetes.io/projected/6893ebba-1b72-4c8e-a877-13904ceb0bfc-kube-api-access-wsflp\") pod \"payload-processing-external-model-6c5884dffd-mcl69\" (UID: \"6893ebba-1b72-4c8e-a877-13904ceb0bfc\") " pod="models-as-a-service/payload-processing-external-model-6c5884dffd-mcl69" Oct 30 18:07:39.773009 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:39.772995 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6893ebba-1b72-4c8e-a877-13904ceb0bfc-provider-credentials\") pod \"payload-processing-external-model-6c5884dffd-mcl69\" (UID: \"6893ebba-1b72-4c8e-a877-13904ceb0bfc\") " pod="models-as-a-service/payload-processing-external-model-6c5884dffd-mcl69" Oct 30 18:07:39.774261 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:39.774234 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6893ebba-1b72-4c8e-a877-13904ceb0bfc-routing-overlay-volume\") pod \"payload-processing-external-model-6c5884dffd-mcl69\" (UID: \"6893ebba-1b72-4c8e-a877-13904ceb0bfc\") " pod="models-as-a-service/payload-processing-external-model-6c5884dffd-mcl69" Oct 30 18:07:39.797200 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:39.796188 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6893ebba-1b72-4c8e-a877-13904ceb0bfc-provider-credentials\") pod \"payload-processing-external-model-6c5884dffd-mcl69\" (UID: \"6893ebba-1b72-4c8e-a877-13904ceb0bfc\") " pod="models-as-a-service/payload-processing-external-model-6c5884dffd-mcl69" Oct 30 18:07:39.871124 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:39.871072 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsflp\" (UniqueName: \"kubernetes.io/projected/6893ebba-1b72-4c8e-a877-13904ceb0bfc-kube-api-access-wsflp\") pod \"payload-processing-external-model-6c5884dffd-mcl69\" (UID: \"6893ebba-1b72-4c8e-a877-13904ceb0bfc\") " pod="models-as-a-service/payload-processing-external-model-6c5884dffd-mcl69" Oct 30 18:07:40.153399 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:40.152628 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c5884dffd-mcl69" Oct 30 18:07:40.203371 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:40.195746 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" Oct 30 18:07:40.203371 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:40.197073 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" Oct 30 18:07:40.228367 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:40.223708 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" Oct 30 18:07:41.381530 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:41.380615 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c5884dffd-mcl69"] Oct 30 18:07:41.384431 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:07:41.384391 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6893ebba_1b72_4c8e_a877_13904ceb0bfc.slice/crio-669ecd5f8ef7bd346d5ce1a36e92a06b76805729c3756c6289694437400ec6d1 WatchSource:0}: Error finding container 669ecd5f8ef7bd346d5ce1a36e92a06b76805729c3756c6289694437400ec6d1: Status 404 returned error can't find the container with id 669ecd5f8ef7bd346d5ce1a36e92a06b76805729c3756c6289694437400ec6d1 Oct 30 18:07:42.153964 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:42.153926 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c5884dffd-mcl69" event={"ID":"6893ebba-1b72-4c8e-a877-13904ceb0bfc","Type":"ContainerStarted","Data":"456487b1380baec4f231faddc63bd9117862b0a9c55f77472fe2994b7365b174"} Oct 30 18:07:42.154175 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:42.153992 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c5884dffd-mcl69" event={"ID":"6893ebba-1b72-4c8e-a877-13904ceb0bfc","Type":"ContainerStarted","Data":"669ecd5f8ef7bd346d5ce1a36e92a06b76805729c3756c6289694437400ec6d1"} Oct 30 18:07:42.154175 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:42.154051 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6c5884dffd-mcl69" Oct 30 18:07:42.201252 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:42.201197 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6c5884dffd-mcl69" podStartSLOduration=3.201182228 podStartE2EDuration="3.201182228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:42.200594512 +0000 UTC m=+914.688231548" watchObservedRunningTime="2026-10-30 18:07:42.201182228 +0000 UTC m=+914.688819262" Oct 30 18:07:54.166631 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:54.166601 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6c5884dffd-mcl69" Oct 30 18:07:54.244245 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:54.244209 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-54fff678bb-bfh89"] Oct 30 18:07:59.244914 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:59.244868 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-54fff678bb-bfh89" podUID="d2fbcb62-dc0f-441a-810c-bb742f7da339" containerName="payload-processing" containerID="cri-o://170f670342e7a7fabccb67eee03fbb5a37f1cec607b62804dc2525f9d326f9a0" gracePeriod=25 Oct 30 18:07:59.526552 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:59.526527 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-54fff678bb-bfh89" Oct 30 18:07:59.634948 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:59.634919 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2fbcb62-dc0f-441a-810c-bb742f7da339-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2fbcb62-dc0f-441a-810c-bb742f7da339-routing-overlay-volume\") pod \"d2fbcb62-dc0f-441a-810c-bb742f7da339\" (UID: \"d2fbcb62-dc0f-441a-810c-bb742f7da339\") " Oct 30 18:07:59.634948 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:59.634962 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2fbcb62-dc0f-441a-810c-bb742f7da339-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2fbcb62-dc0f-441a-810c-bb742f7da339-provider-credentials\") pod \"d2fbcb62-dc0f-441a-810c-bb742f7da339\" (UID: \"d2fbcb62-dc0f-441a-810c-bb742f7da339\") " Oct 30 18:07:59.635220 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:59.635000 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2fbcb62-dc0f-441a-810c-bb742f7da339-kube-api-access-kjbmj\" (UniqueName: \"kubernetes.io/projected/d2fbcb62-dc0f-441a-810c-bb742f7da339-kube-api-access-kjbmj\") pod \"d2fbcb62-dc0f-441a-810c-bb742f7da339\" (UID: \"d2fbcb62-dc0f-441a-810c-bb742f7da339\") " Oct 30 18:07:59.635220 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:59.635163 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2fbcb62-dc0f-441a-810c-bb742f7da339-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2fbcb62-dc0f-441a-810c-bb742f7da339-plugins-config-volume\") pod \"d2fbcb62-dc0f-441a-810c-bb742f7da339\" (UID: \"d2fbcb62-dc0f-441a-810c-bb742f7da339\") " Oct 30 18:07:59.635360 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:59.635308 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2fbcb62-dc0f-441a-810c-bb742f7da339-routing-overlay-volume" pod "d2fbcb62-dc0f-441a-810c-bb742f7da339" (UID: "d2fbcb62-dc0f-441a-810c-bb742f7da339"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:07:59.635623 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:59.635596 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2fbcb62-dc0f-441a-810c-bb742f7da339-plugins-config-volume" pod "d2fbcb62-dc0f-441a-810c-bb742f7da339" (UID: "d2fbcb62-dc0f-441a-810c-bb742f7da339"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:07:59.635727 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:59.635635 4455 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2fbcb62-dc0f-441a-810c-bb742f7da339-routing-overlay-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:07:59.637813 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:59.637786 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2fbcb62-dc0f-441a-810c-bb742f7da339-kube-api-access-kjbmj" pod "d2fbcb62-dc0f-441a-810c-bb742f7da339" (UID: "d2fbcb62-dc0f-441a-810c-bb742f7da339"). InnerVolumeSpecName "kube-api-access-kjbmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:07:59.637904 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:59.637833 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2fbcb62-dc0f-441a-810c-bb742f7da339-provider-credentials" pod "d2fbcb62-dc0f-441a-810c-bb742f7da339" (UID: "d2fbcb62-dc0f-441a-810c-bb742f7da339"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:07:59.736467 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:59.736439 4455 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2fbcb62-dc0f-441a-810c-bb742f7da339-provider-credentials\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:07:59.736467 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:59.736463 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjbmj\" (UniqueName: \"kubernetes.io/projected/d2fbcb62-dc0f-441a-810c-bb742f7da339-kube-api-access-kjbmj\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:07:59.736667 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:07:59.736485 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2fbcb62-dc0f-441a-810c-bb742f7da339-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:08:00.289885 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:00.289817 4455 generic.go:356] "Generic (PLEG): container finished" podID="d2fbcb62-dc0f-441a-810c-bb742f7da339" containerID="170f670342e7a7fabccb67eee03fbb5a37f1cec607b62804dc2525f9d326f9a0" exitCode=0 Oct 30 18:08:00.289885 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:00.289897 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-54fff678bb-bfh89" Oct 30 18:08:00.290525 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:00.289976 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-54fff678bb-bfh89" event={"ID":"d2fbcb62-dc0f-441a-810c-bb742f7da339","Type":"ContainerDied","Data":"170f670342e7a7fabccb67eee03fbb5a37f1cec607b62804dc2525f9d326f9a0"} Oct 30 18:08:00.290525 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:00.290026 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-54fff678bb-bfh89" event={"ID":"d2fbcb62-dc0f-441a-810c-bb742f7da339","Type":"ContainerDied","Data":"1ea9ed387f35fda284033d73aa654bc6001715d55d8429f2136584eee1dbe9a8"} Oct 30 18:08:00.290525 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:00.290063 4455 scope.go:122] "RemoveContainer" containerID="170f670342e7a7fabccb67eee03fbb5a37f1cec607b62804dc2525f9d326f9a0" Oct 30 18:08:00.312507 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:00.312485 4455 scope.go:122] "RemoveContainer" containerID="170f670342e7a7fabccb67eee03fbb5a37f1cec607b62804dc2525f9d326f9a0" Oct 30 18:08:00.312867 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:08:00.312847 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"170f670342e7a7fabccb67eee03fbb5a37f1cec607b62804dc2525f9d326f9a0\": container with ID starting with 170f670342e7a7fabccb67eee03fbb5a37f1cec607b62804dc2525f9d326f9a0 not found: ID does not exist" containerID="170f670342e7a7fabccb67eee03fbb5a37f1cec607b62804dc2525f9d326f9a0" Oct 30 18:08:00.312914 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:00.312878 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"170f670342e7a7fabccb67eee03fbb5a37f1cec607b62804dc2525f9d326f9a0"} err="failed to get container status \"170f670342e7a7fabccb67eee03fbb5a37f1cec607b62804dc2525f9d326f9a0\": rpc error: code = NotFound desc = could not find container \"170f670342e7a7fabccb67eee03fbb5a37f1cec607b62804dc2525f9d326f9a0\": container with ID starting with 170f670342e7a7fabccb67eee03fbb5a37f1cec607b62804dc2525f9d326f9a0 not found: ID does not exist" Oct 30 18:08:00.352996 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:00.352964 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-54fff678bb-bfh89"] Oct 30 18:08:00.367394 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:00.367299 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-54fff678bb-bfh89"] Oct 30 18:08:00.385622 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:08:00.385588 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:08:01.635605 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:01.635569 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2fbcb62-dc0f-441a-810c-bb742f7da339" path="/var/lib/kubelet/pods/d2fbcb62-dc0f-441a-810c-bb742f7da339/volumes" Oct 30 18:08:04.609774 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:04.609743 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd"] Oct 30 18:08:04.610577 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:04.610555 4455 cpu_manager.go:414] "containerMap: removing container" podUID="d2fbcb62-dc0f-441a-810c-bb742f7da339" containerName="payload-processing" Oct 30 18:08:04.610577 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:04.610576 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2fbcb62-dc0f-441a-810c-bb742f7da339" containerName="payload-processing" Oct 30 18:08:04.610747 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:04.610706 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2fbcb62-dc0f-441a-810c-bb742f7da339" containerName="payload-processing" Oct 30 18:08:04.615707 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:04.615688 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" Oct 30 18:08:04.640218 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:04.640192 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd"] Oct 30 18:08:04.713662 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:04.713621 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d3471dc-aaf3-4060-a64d-2413e522e65a-istio-envoy\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" Oct 30 18:08:04.713846 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:04.713675 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d3471dc-aaf3-4060-a64d-2413e522e65a-workload-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" Oct 30 18:08:04.713846 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:04.713732 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d3471dc-aaf3-4060-a64d-2413e522e65a-istio-token\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" Oct 30 18:08:04.714528 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:04.714495 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d3471dc-aaf3-4060-a64d-2413e522e65a-istio-data\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" Oct 30 18:08:04.714651 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:04.714539 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d3471dc-aaf3-4060-a64d-2413e522e65a-istio-podinfo\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" Oct 30 18:08:04.714651 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:04.714563 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d3471dc-aaf3-4060-a64d-2413e522e65a-istiod-ca-cert\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" Oct 30 18:08:04.714651 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:04.714631 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r4ff\" (UniqueName: \"kubernetes.io/projected/1d3471dc-aaf3-4060-a64d-2413e522e65a-kube-api-access-5r4ff\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" Oct 30 18:08:04.714824 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:04.714701 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d3471dc-aaf3-4060-a64d-2413e522e65a-credential-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" Oct 30 18:08:04.714824 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:04.714761 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d3471dc-aaf3-4060-a64d-2413e522e65a-workload-certs\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" Oct 30 18:08:04.816594 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:04.816554 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r4ff\" (UniqueName: \"kubernetes.io/projected/1d3471dc-aaf3-4060-a64d-2413e522e65a-kube-api-access-5r4ff\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" Oct 30 18:08:04.816759 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:04.816618 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d3471dc-aaf3-4060-a64d-2413e522e65a-credential-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" Oct 30 18:08:04.816759 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:04.816674 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d3471dc-aaf3-4060-a64d-2413e522e65a-workload-certs\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" Oct 30 18:08:04.816877 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:04.816792 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d3471dc-aaf3-4060-a64d-2413e522e65a-istio-envoy\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" Oct 30 18:08:04.816877 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:04.816822 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d3471dc-aaf3-4060-a64d-2413e522e65a-workload-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" Oct 30 18:08:04.816987 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:04.816880 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d3471dc-aaf3-4060-a64d-2413e522e65a-istio-token\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" Oct 30 18:08:04.816987 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:04.816965 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d3471dc-aaf3-4060-a64d-2413e522e65a-istio-data\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" Oct 30 18:08:04.817093 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:04.816994 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d3471dc-aaf3-4060-a64d-2413e522e65a-istio-podinfo\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" Oct 30 18:08:04.817093 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:04.817017 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d3471dc-aaf3-4060-a64d-2413e522e65a-istiod-ca-cert\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" Oct 30 18:08:04.817271 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:04.817083 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d3471dc-aaf3-4060-a64d-2413e522e65a-credential-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" Oct 30 18:08:04.817271 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:04.817105 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d3471dc-aaf3-4060-a64d-2413e522e65a-workload-certs\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" Oct 30 18:08:04.817444 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:04.817284 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d3471dc-aaf3-4060-a64d-2413e522e65a-workload-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" Oct 30 18:08:04.817444 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:04.817381 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d3471dc-aaf3-4060-a64d-2413e522e65a-istio-data\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" Oct 30 18:08:04.817754 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:04.817729 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d3471dc-aaf3-4060-a64d-2413e522e65a-istiod-ca-cert\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" Oct 30 18:08:04.819574 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:04.819548 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d3471dc-aaf3-4060-a64d-2413e522e65a-istio-envoy\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" Oct 30 18:08:04.819691 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:04.819645 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d3471dc-aaf3-4060-a64d-2413e522e65a-istio-podinfo\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" Oct 30 18:08:04.842183 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:04.842158 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d3471dc-aaf3-4060-a64d-2413e522e65a-istio-token\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" Oct 30 18:08:04.842314 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:04.842296 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r4ff\" (UniqueName: \"kubernetes.io/projected/1d3471dc-aaf3-4060-a64d-2413e522e65a-kube-api-access-5r4ff\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" Oct 30 18:08:04.933694 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:04.933606 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" Oct 30 18:08:05.471984 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:05.471957 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd"] Oct 30 18:08:05.472099 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:08:05.471955 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d3471dc_aaf3_4060_a64d_2413e522e65a.slice/crio-3ea70c37d22d111a79b3374f1cc119caefe01505d709500a8cf0f74869fa5c0e WatchSource:0}: Error finding container 3ea70c37d22d111a79b3374f1cc119caefe01505d709500a8cf0f74869fa5c0e: Status 404 returned error can't find the container with id 3ea70c37d22d111a79b3374f1cc119caefe01505d709500a8cf0f74869fa5c0e Oct 30 18:08:05.474991 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:05.474954 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:08:05.475075 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:05.475040 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:08:05.475122 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:05.475079 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:08:06.340089 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:06.339144 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" event={"ID":"1d3471dc-aaf3-4060-a64d-2413e522e65a","Type":"ContainerStarted","Data":"af56ed4e6bb5f101fe282f280b2506a9145a50eefe1f248c8a104b25b4ae0d67"} Oct 30 18:08:06.340089 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:06.339198 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" event={"ID":"1d3471dc-aaf3-4060-a64d-2413e522e65a","Type":"ContainerStarted","Data":"3ea70c37d22d111a79b3374f1cc119caefe01505d709500a8cf0f74869fa5c0e"} Oct 30 18:08:06.509438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:06.509369 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" podStartSLOduration=2.509311637 podStartE2EDuration="2.509311637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:08:06.41064943 +0000 UTC m=+938.898286479" watchObservedRunningTime="2026-10-30 18:08:06.509311637 +0000 UTC m=+938.996948673" Oct 30 18:08:06.511381 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:06.511351 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-78d8ccc49c-8txzp"] Oct 30 18:08:06.516781 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:06.516758 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78d8ccc49c-8txzp" Oct 30 18:08:06.524705 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:06.524334 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-78d8ccc49c-8txzp"] Oct 30 18:08:06.532751 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:06.532725 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0b1e9d0-b896-4a83-844d-d566d3d324d6-provider-credentials\") pod \"payload-processing-external-model-78d8ccc49c-8txzp\" (UID: \"f0b1e9d0-b896-4a83-844d-d566d3d324d6\") " pod="models-as-a-service/payload-processing-external-model-78d8ccc49c-8txzp" Oct 30 18:08:06.532881 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:06.532768 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0b1e9d0-b896-4a83-844d-d566d3d324d6-routing-overlay-volume\") pod \"payload-processing-external-model-78d8ccc49c-8txzp\" (UID: \"f0b1e9d0-b896-4a83-844d-d566d3d324d6\") " pod="models-as-a-service/payload-processing-external-model-78d8ccc49c-8txzp" Oct 30 18:08:06.532881 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:06.532788 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6vqw\" (UniqueName: \"kubernetes.io/projected/f0b1e9d0-b896-4a83-844d-d566d3d324d6-kube-api-access-r6vqw\") pod \"payload-processing-external-model-78d8ccc49c-8txzp\" (UID: \"f0b1e9d0-b896-4a83-844d-d566d3d324d6\") " pod="models-as-a-service/payload-processing-external-model-78d8ccc49c-8txzp" Oct 30 18:08:06.532993 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:06.532891 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0b1e9d0-b896-4a83-844d-d566d3d324d6-plugins-config-volume\") pod \"payload-processing-external-model-78d8ccc49c-8txzp\" (UID: \"f0b1e9d0-b896-4a83-844d-d566d3d324d6\") " pod="models-as-a-service/payload-processing-external-model-78d8ccc49c-8txzp" Oct 30 18:08:06.533972 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:06.533948 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-33f4c67a6439\"" Oct 30 18:08:06.534067 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:06.534022 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1c197d0c785d\"" Oct 30 18:08:06.634625 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:06.634519 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0b1e9d0-b896-4a83-844d-d566d3d324d6-provider-credentials\") pod \"payload-processing-external-model-78d8ccc49c-8txzp\" (UID: \"f0b1e9d0-b896-4a83-844d-d566d3d324d6\") " pod="models-as-a-service/payload-processing-external-model-78d8ccc49c-8txzp" Oct 30 18:08:06.634625 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:06.634575 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0b1e9d0-b896-4a83-844d-d566d3d324d6-routing-overlay-volume\") pod \"payload-processing-external-model-78d8ccc49c-8txzp\" (UID: \"f0b1e9d0-b896-4a83-844d-d566d3d324d6\") " pod="models-as-a-service/payload-processing-external-model-78d8ccc49c-8txzp" Oct 30 18:08:06.634625 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:06.634593 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6vqw\" (UniqueName: \"kubernetes.io/projected/f0b1e9d0-b896-4a83-844d-d566d3d324d6-kube-api-access-r6vqw\") pod \"payload-processing-external-model-78d8ccc49c-8txzp\" (UID: \"f0b1e9d0-b896-4a83-844d-d566d3d324d6\") " pod="models-as-a-service/payload-processing-external-model-78d8ccc49c-8txzp" Oct 30 18:08:06.634925 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:06.634682 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0b1e9d0-b896-4a83-844d-d566d3d324d6-plugins-config-volume\") pod \"payload-processing-external-model-78d8ccc49c-8txzp\" (UID: \"f0b1e9d0-b896-4a83-844d-d566d3d324d6\") " pod="models-as-a-service/payload-processing-external-model-78d8ccc49c-8txzp" Oct 30 18:08:06.635388 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:06.635366 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0b1e9d0-b896-4a83-844d-d566d3d324d6-plugins-config-volume\") pod \"payload-processing-external-model-78d8ccc49c-8txzp\" (UID: \"f0b1e9d0-b896-4a83-844d-d566d3d324d6\") " pod="models-as-a-service/payload-processing-external-model-78d8ccc49c-8txzp" Oct 30 18:08:06.636053 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:06.636031 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0b1e9d0-b896-4a83-844d-d566d3d324d6-routing-overlay-volume\") pod \"payload-processing-external-model-78d8ccc49c-8txzp\" (UID: \"f0b1e9d0-b896-4a83-844d-d566d3d324d6\") " pod="models-as-a-service/payload-processing-external-model-78d8ccc49c-8txzp" Oct 30 18:08:06.639044 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:06.639019 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0b1e9d0-b896-4a83-844d-d566d3d324d6-provider-credentials\") pod \"payload-processing-external-model-78d8ccc49c-8txzp\" (UID: \"f0b1e9d0-b896-4a83-844d-d566d3d324d6\") " pod="models-as-a-service/payload-processing-external-model-78d8ccc49c-8txzp" Oct 30 18:08:06.662012 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:06.661975 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6vqw\" (UniqueName: \"kubernetes.io/projected/f0b1e9d0-b896-4a83-844d-d566d3d324d6-kube-api-access-r6vqw\") pod \"payload-processing-external-model-78d8ccc49c-8txzp\" (UID: \"f0b1e9d0-b896-4a83-844d-d566d3d324d6\") " pod="models-as-a-service/payload-processing-external-model-78d8ccc49c-8txzp" Oct 30 18:08:06.836947 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:06.836909 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78d8ccc49c-8txzp" Oct 30 18:08:06.934808 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:06.934777 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" Oct 30 18:08:06.936862 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:06.936522 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:08:06.936862 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:06.936575 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" podUID="1d3471dc-aaf3-4060-a64d-2413e522e65a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 18:08:07.203491 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:07.201912 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-78d8ccc49c-8txzp"] Oct 30 18:08:07.350820 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:07.350784 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78d8ccc49c-8txzp" event={"ID":"f0b1e9d0-b896-4a83-844d-d566d3d324d6","Type":"ContainerStarted","Data":"046b6dd409fb81acd39cd06f84189bbf263e7e0165243c376f25005134f786ec"} Oct 30 18:08:07.351216 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:07.351021 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-78d8ccc49c-8txzp" Oct 30 18:08:07.351216 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:07.351047 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78d8ccc49c-8txzp" event={"ID":"f0b1e9d0-b896-4a83-844d-d566d3d324d6","Type":"ContainerStarted","Data":"6a059a03a2f3f09c579f3eabe457d73e7f3bd36bbed1117d0312d13975c14883"} Oct 30 18:08:07.412467 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:07.412413 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-78d8ccc49c-8txzp" podStartSLOduration=1.4123977189999999 podStartE2EDuration="1.412397719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:08:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:08:07.411475848 +0000 UTC m=+939.899112883" watchObservedRunningTime="2026-10-30 18:08:07.412397719 +0000 UTC m=+939.900034751" Oct 30 18:08:07.934203 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:07.934169 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:08:07.934416 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:07.934226 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" podUID="1d3471dc-aaf3-4060-a64d-2413e522e65a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 18:08:08.934173 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:08.934079 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:08:08.934173 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:08.934160 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" podUID="1d3471dc-aaf3-4060-a64d-2413e522e65a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 18:08:09.936913 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:09.936876 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:08:09.940445 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:09.940407 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" podUID="1d3471dc-aaf3-4060-a64d-2413e522e65a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 18:08:10.936347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:10.935990 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:08:10.936347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:10.936057 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" podUID="1d3471dc-aaf3-4060-a64d-2413e522e65a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 18:08:11.935345 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:11.934677 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:08:11.935345 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:11.934740 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" podUID="1d3471dc-aaf3-4060-a64d-2413e522e65a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 18:08:12.944457 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:12.942860 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:08:12.944457 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:12.942920 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" podUID="1d3471dc-aaf3-4060-a64d-2413e522e65a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 18:08:13.625470 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:13.622847 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:08:13.625470 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:13.622946 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" podUID="c7c21be6-b9eb-401c-aa3c-af5b62960cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:13.936405 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:13.936054 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:08:13.936405 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:13.936149 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" podUID="1d3471dc-aaf3-4060-a64d-2413e522e65a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 18:08:13.993354 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:13.991469 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:08:13.993354 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:13.991560 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" podUID="4e7b0c73-0041-4549-b47e-76a7aaed2b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:15.163936 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:15.163903 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" Oct 30 18:08:15.164466 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:15.164439 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" Oct 30 18:08:15.165435 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:15.165413 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" Oct 30 18:08:18.363159 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:18.363132 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-78d8ccc49c-8txzp" Oct 30 18:08:18.444661 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:18.444614 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c5884dffd-mcl69"] Oct 30 18:08:23.445448 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:23.445376 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6c5884dffd-mcl69" podUID="6893ebba-1b72-4c8e-a877-13904ceb0bfc" containerName="payload-processing" containerID="cri-o://456487b1380baec4f231faddc63bd9117862b0a9c55f77472fe2994b7365b174" gracePeriod=25 Oct 30 18:08:23.677282 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:23.677251 4455 generic.go:356] "Generic (PLEG): container finished" podID="6893ebba-1b72-4c8e-a877-13904ceb0bfc" containerID="456487b1380baec4f231faddc63bd9117862b0a9c55f77472fe2994b7365b174" exitCode=0 Oct 30 18:08:23.677478 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:23.677361 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c5884dffd-mcl69" event={"ID":"6893ebba-1b72-4c8e-a877-13904ceb0bfc","Type":"ContainerDied","Data":"456487b1380baec4f231faddc63bd9117862b0a9c55f77472fe2994b7365b174"} Oct 30 18:08:23.735579 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:23.735545 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c5884dffd-mcl69" Oct 30 18:08:23.802163 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:23.802127 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6893ebba-1b72-4c8e-a877-13904ceb0bfc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6893ebba-1b72-4c8e-a877-13904ceb0bfc-routing-overlay-volume\") pod \"6893ebba-1b72-4c8e-a877-13904ceb0bfc\" (UID: \"6893ebba-1b72-4c8e-a877-13904ceb0bfc\") " Oct 30 18:08:23.802377 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:23.802245 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6893ebba-1b72-4c8e-a877-13904ceb0bfc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6893ebba-1b72-4c8e-a877-13904ceb0bfc-plugins-config-volume\") pod \"6893ebba-1b72-4c8e-a877-13904ceb0bfc\" (UID: \"6893ebba-1b72-4c8e-a877-13904ceb0bfc\") " Oct 30 18:08:23.802454 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:23.802390 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6893ebba-1b72-4c8e-a877-13904ceb0bfc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6893ebba-1b72-4c8e-a877-13904ceb0bfc-provider-credentials\") pod \"6893ebba-1b72-4c8e-a877-13904ceb0bfc\" (UID: \"6893ebba-1b72-4c8e-a877-13904ceb0bfc\") " Oct 30 18:08:23.802547 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:23.802533 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6893ebba-1b72-4c8e-a877-13904ceb0bfc-kube-api-access-wsflp\" (UniqueName: \"kubernetes.io/projected/6893ebba-1b72-4c8e-a877-13904ceb0bfc-kube-api-access-wsflp\") pod \"6893ebba-1b72-4c8e-a877-13904ceb0bfc\" (UID: \"6893ebba-1b72-4c8e-a877-13904ceb0bfc\") " Oct 30 18:08:23.802624 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:23.802597 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6893ebba-1b72-4c8e-a877-13904ceb0bfc-routing-overlay-volume" pod "6893ebba-1b72-4c8e-a877-13904ceb0bfc" (UID: "6893ebba-1b72-4c8e-a877-13904ceb0bfc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:08:23.802725 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:23.802703 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6893ebba-1b72-4c8e-a877-13904ceb0bfc-plugins-config-volume" pod "6893ebba-1b72-4c8e-a877-13904ceb0bfc" (UID: "6893ebba-1b72-4c8e-a877-13904ceb0bfc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:08:23.803054 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:23.803015 4455 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6893ebba-1b72-4c8e-a877-13904ceb0bfc-routing-overlay-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:08:23.803054 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:23.803042 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6893ebba-1b72-4c8e-a877-13904ceb0bfc-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:08:23.805030 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:23.804999 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6893ebba-1b72-4c8e-a877-13904ceb0bfc-provider-credentials" pod "6893ebba-1b72-4c8e-a877-13904ceb0bfc" (UID: "6893ebba-1b72-4c8e-a877-13904ceb0bfc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:08:23.805254 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:23.805230 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6893ebba-1b72-4c8e-a877-13904ceb0bfc-kube-api-access-wsflp" pod "6893ebba-1b72-4c8e-a877-13904ceb0bfc" (UID: "6893ebba-1b72-4c8e-a877-13904ceb0bfc"). InnerVolumeSpecName "kube-api-access-wsflp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:08:23.905156 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:23.905118 4455 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6893ebba-1b72-4c8e-a877-13904ceb0bfc-provider-credentials\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:08:23.905156 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:23.905156 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsflp\" (UniqueName: \"kubernetes.io/projected/6893ebba-1b72-4c8e-a877-13904ceb0bfc-kube-api-access-wsflp\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:08:24.513986 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.513949 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp"] Oct 30 18:08:24.515489 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.515458 4455 cpu_manager.go:414] "containerMap: removing container" podUID="6893ebba-1b72-4c8e-a877-13904ceb0bfc" containerName="payload-processing" Oct 30 18:08:24.515489 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.515486 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6893ebba-1b72-4c8e-a877-13904ceb0bfc" containerName="payload-processing" Oct 30 18:08:24.515746 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.515730 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="6893ebba-1b72-4c8e-a877-13904ceb0bfc" containerName="payload-processing" Oct 30 18:08:24.520637 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.520614 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" Oct 30 18:08:24.547012 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.546982 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp"] Oct 30 18:08:24.613287 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.613249 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac80ccea-a611-4de1-951c-479ad3d827a7-credential-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" Oct 30 18:08:24.613518 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.613293 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac80ccea-a611-4de1-951c-479ad3d827a7-istiod-ca-cert\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" Oct 30 18:08:24.613518 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.613376 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac80ccea-a611-4de1-951c-479ad3d827a7-workload-certs\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" Oct 30 18:08:24.613518 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.613478 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac80ccea-a611-4de1-951c-479ad3d827a7-workload-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" Oct 30 18:08:24.613670 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.613588 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac80ccea-a611-4de1-951c-479ad3d827a7-istio-data\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" Oct 30 18:08:24.613670 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.613632 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac80ccea-a611-4de1-951c-479ad3d827a7-istio-podinfo\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" Oct 30 18:08:24.613670 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.613649 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dknf\" (UniqueName: \"kubernetes.io/projected/ac80ccea-a611-4de1-951c-479ad3d827a7-kube-api-access-5dknf\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" Oct 30 18:08:24.613826 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.613674 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac80ccea-a611-4de1-951c-479ad3d827a7-istio-token\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" Oct 30 18:08:24.613826 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.613702 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac80ccea-a611-4de1-951c-479ad3d827a7-istio-envoy\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" Oct 30 18:08:24.686525 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.686484 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c5884dffd-mcl69" event={"ID":"6893ebba-1b72-4c8e-a877-13904ceb0bfc","Type":"ContainerDied","Data":"669ecd5f8ef7bd346d5ce1a36e92a06b76805729c3756c6289694437400ec6d1"} Oct 30 18:08:24.686712 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.686549 4455 scope.go:122] "RemoveContainer" containerID="456487b1380baec4f231faddc63bd9117862b0a9c55f77472fe2994b7365b174" Oct 30 18:08:24.686712 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.686501 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c5884dffd-mcl69" Oct 30 18:08:24.722670 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.722631 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac80ccea-a611-4de1-951c-479ad3d827a7-istio-data\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" Oct 30 18:08:24.722859 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.722710 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac80ccea-a611-4de1-951c-479ad3d827a7-istio-podinfo\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" Oct 30 18:08:24.722859 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.722739 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dknf\" (UniqueName: \"kubernetes.io/projected/ac80ccea-a611-4de1-951c-479ad3d827a7-kube-api-access-5dknf\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" Oct 30 18:08:24.722859 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.722804 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac80ccea-a611-4de1-951c-479ad3d827a7-istio-token\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" Oct 30 18:08:24.723099 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.722864 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac80ccea-a611-4de1-951c-479ad3d827a7-istio-envoy\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" Oct 30 18:08:24.723099 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.723030 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac80ccea-a611-4de1-951c-479ad3d827a7-credential-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" Oct 30 18:08:24.723099 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.723063 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac80ccea-a611-4de1-951c-479ad3d827a7-istiod-ca-cert\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" Oct 30 18:08:24.723258 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.723133 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac80ccea-a611-4de1-951c-479ad3d827a7-workload-certs\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" Oct 30 18:08:24.723258 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.723195 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac80ccea-a611-4de1-951c-479ad3d827a7-workload-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" Oct 30 18:08:24.723764 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.723735 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac80ccea-a611-4de1-951c-479ad3d827a7-workload-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" Oct 30 18:08:24.724029 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.724006 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac80ccea-a611-4de1-951c-479ad3d827a7-istio-data\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" Oct 30 18:08:24.731304 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.731275 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac80ccea-a611-4de1-951c-479ad3d827a7-istio-podinfo\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" Oct 30 18:08:24.731642 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.731617 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac80ccea-a611-4de1-951c-479ad3d827a7-credential-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" Oct 30 18:08:24.732244 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.732214 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac80ccea-a611-4de1-951c-479ad3d827a7-istiod-ca-cert\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" Oct 30 18:08:24.732525 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.732503 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac80ccea-a611-4de1-951c-479ad3d827a7-workload-certs\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" Oct 30 18:08:24.747639 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.747602 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac80ccea-a611-4de1-951c-479ad3d827a7-istio-envoy\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" Oct 30 18:08:24.749804 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.749780 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c5884dffd-mcl69"] Oct 30 18:08:24.759023 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.758968 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c5884dffd-mcl69"] Oct 30 18:08:24.759753 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.759732 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac80ccea-a611-4de1-951c-479ad3d827a7-istio-token\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" Oct 30 18:08:24.759920 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.759896 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dknf\" (UniqueName: \"kubernetes.io/projected/ac80ccea-a611-4de1-951c-479ad3d827a7-kube-api-access-5dknf\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" Oct 30 18:08:24.763308 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.763283 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584"] Oct 30 18:08:24.769546 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.769489 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" Oct 30 18:08:24.807930 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.807894 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584"] Oct 30 18:08:24.837369 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.837312 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" Oct 30 18:08:24.927149 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.927115 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7de539f-17e9-453c-af6b-e83fc7c91239-istio-token\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" Oct 30 18:08:24.927438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.927153 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7de539f-17e9-453c-af6b-e83fc7c91239-workload-certs\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" Oct 30 18:08:24.927438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.927187 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqbm6\" (UniqueName: \"kubernetes.io/projected/d7de539f-17e9-453c-af6b-e83fc7c91239-kube-api-access-jqbm6\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" Oct 30 18:08:24.927587 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.927458 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7de539f-17e9-453c-af6b-e83fc7c91239-credential-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" Oct 30 18:08:24.927646 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.927599 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7de539f-17e9-453c-af6b-e83fc7c91239-istiod-ca-cert\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" Oct 30 18:08:24.927807 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.927788 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7de539f-17e9-453c-af6b-e83fc7c91239-istio-podinfo\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" Oct 30 18:08:24.928049 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.928023 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7de539f-17e9-453c-af6b-e83fc7c91239-istio-envoy\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" Oct 30 18:08:24.928150 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.928130 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7de539f-17e9-453c-af6b-e83fc7c91239-istio-data\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" Oct 30 18:08:24.928365 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:24.928313 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7de539f-17e9-453c-af6b-e83fc7c91239-workload-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" Oct 30 18:08:25.030317 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:25.030221 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7de539f-17e9-453c-af6b-e83fc7c91239-credential-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" Oct 30 18:08:25.030317 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:25.030272 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7de539f-17e9-453c-af6b-e83fc7c91239-istiod-ca-cert\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" Oct 30 18:08:25.030597 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:25.030477 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7de539f-17e9-453c-af6b-e83fc7c91239-istio-podinfo\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" Oct 30 18:08:25.030597 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:25.030515 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7de539f-17e9-453c-af6b-e83fc7c91239-istio-envoy\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" Oct 30 18:08:25.030696 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:25.030594 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7de539f-17e9-453c-af6b-e83fc7c91239-istio-data\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" Oct 30 18:08:25.030696 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:25.030637 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7de539f-17e9-453c-af6b-e83fc7c91239-credential-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" Oct 30 18:08:25.030696 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:25.030666 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7de539f-17e9-453c-af6b-e83fc7c91239-workload-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" Oct 30 18:08:25.030846 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:25.030709 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7de539f-17e9-453c-af6b-e83fc7c91239-istio-token\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" Oct 30 18:08:25.030846 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:25.030744 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7de539f-17e9-453c-af6b-e83fc7c91239-workload-certs\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" Oct 30 18:08:25.030846 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:25.030777 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqbm6\" (UniqueName: \"kubernetes.io/projected/d7de539f-17e9-453c-af6b-e83fc7c91239-kube-api-access-jqbm6\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" Oct 30 18:08:25.030993 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:25.030936 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7de539f-17e9-453c-af6b-e83fc7c91239-istio-data\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" Oct 30 18:08:25.031035 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:25.030993 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7de539f-17e9-453c-af6b-e83fc7c91239-istiod-ca-cert\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" Oct 30 18:08:25.031355 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:25.031314 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7de539f-17e9-453c-af6b-e83fc7c91239-workload-certs\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" Oct 30 18:08:25.031459 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:25.031437 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7de539f-17e9-453c-af6b-e83fc7c91239-workload-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" Oct 30 18:08:25.033577 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:25.033548 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7de539f-17e9-453c-af6b-e83fc7c91239-istio-envoy\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" Oct 30 18:08:25.033745 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:25.033730 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7de539f-17e9-453c-af6b-e83fc7c91239-istio-podinfo\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" Oct 30 18:08:25.056186 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:25.056156 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7de539f-17e9-453c-af6b-e83fc7c91239-istio-token\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" Oct 30 18:08:25.056284 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:25.056231 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqbm6\" (UniqueName: \"kubernetes.io/projected/d7de539f-17e9-453c-af6b-e83fc7c91239-kube-api-access-jqbm6\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" Oct 30 18:08:25.112572 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:25.112543 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" Oct 30 18:08:25.163958 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:25.163935 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp"] Oct 30 18:08:25.164210 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:25.164182 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:08:25.164256 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:25.164243 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:08:25.164294 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:25.164276 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:08:25.436799 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:08:25.436767 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7de539f_17e9_453c_af6b_e83fc7c91239.slice/crio-5f5ad7dbc5f2d56205d26b118b0a0506f969fe95aa0d974c1d72d7fada4b9885 WatchSource:0}: Error finding container 5f5ad7dbc5f2d56205d26b118b0a0506f969fe95aa0d974c1d72d7fada4b9885: Status 404 returned error can't find the container with id 5f5ad7dbc5f2d56205d26b118b0a0506f969fe95aa0d974c1d72d7fada4b9885 Oct 30 18:08:25.437252 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:25.437226 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584"] Oct 30 18:08:25.439729 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:25.439693 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:08:25.439813 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:25.439763 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:08:25.439813 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:25.439793 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:08:25.636762 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:25.636724 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6893ebba-1b72-4c8e-a877-13904ceb0bfc" path="/var/lib/kubelet/pods/6893ebba-1b72-4c8e-a877-13904ceb0bfc/volumes" Oct 30 18:08:25.698206 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:25.698107 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" event={"ID":"ac80ccea-a611-4de1-951c-479ad3d827a7","Type":"ContainerStarted","Data":"36e7d4a4bb7018d8514f41830ca5589fbd87445555b98301f09584b942a62f0f"} Oct 30 18:08:25.698206 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:25.698176 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" event={"ID":"ac80ccea-a611-4de1-951c-479ad3d827a7","Type":"ContainerStarted","Data":"06b57ac3c4ad611959006b0fe39972cde4f9b5382652d440f5ecb84b6d98ba8e"} Oct 30 18:08:25.700376 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:25.700349 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" event={"ID":"d7de539f-17e9-453c-af6b-e83fc7c91239","Type":"ContainerStarted","Data":"4c0afb4f15150e31f8a16b39c86d19d0b16977cddbc40c5b5cf4debe0e59c567"} Oct 30 18:08:25.700497 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:25.700403 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" event={"ID":"d7de539f-17e9-453c-af6b-e83fc7c91239","Type":"ContainerStarted","Data":"5f5ad7dbc5f2d56205d26b118b0a0506f969fe95aa0d974c1d72d7fada4b9885"} Oct 30 18:08:25.825010 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:25.824955 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" podStartSLOduration=1.824940096 podStartE2EDuration="1.824940096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:08:25.753782985 +0000 UTC m=+958.241420020" watchObservedRunningTime="2026-10-30 18:08:25.824940096 +0000 UTC m=+958.312577135" Oct 30 18:08:25.827145 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:25.827102 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" podStartSLOduration=1.827089929 podStartE2EDuration="1.827089929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:08:25.823890259 +0000 UTC m=+958.311527294" watchObservedRunningTime="2026-10-30 18:08:25.827089929 +0000 UTC m=+958.314726941" Oct 30 18:08:25.837754 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:25.837729 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" Oct 30 18:08:26.114262 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:26.113707 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" Oct 30 18:08:26.115402 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:26.115373 4455 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 18:08:26.115547 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:26.115424 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" podUID="d7de539f-17e9-453c-af6b-e83fc7c91239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 18:08:26.839119 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:26.839076 4455 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 18:08:26.839553 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:26.839137 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" podUID="ac80ccea-a611-4de1-951c-479ad3d827a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 18:08:27.114405 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:27.113720 4455 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 18:08:27.114405 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:27.113784 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" podUID="d7de539f-17e9-453c-af6b-e83fc7c91239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 18:08:27.837903 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:27.837872 4455 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 18:08:27.838081 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:27.837928 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" podUID="ac80ccea-a611-4de1-951c-479ad3d827a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 18:08:28.113782 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:28.113691 4455 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 18:08:28.114378 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:28.114344 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" podUID="d7de539f-17e9-453c-af6b-e83fc7c91239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 18:08:28.838106 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:28.838069 4455 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 18:08:28.838261 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:28.838134 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" podUID="ac80ccea-a611-4de1-951c-479ad3d827a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 18:08:29.117433 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:29.116816 4455 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 18:08:29.117433 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:29.116895 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" podUID="d7de539f-17e9-453c-af6b-e83fc7c91239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 18:08:29.839100 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:29.839055 4455 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 18:08:29.839487 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:29.839441 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" podUID="ac80ccea-a611-4de1-951c-479ad3d827a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 18:08:30.115926 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:30.115831 4455 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 18:08:30.116303 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:30.116271 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" podUID="d7de539f-17e9-453c-af6b-e83fc7c91239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 18:08:30.440248 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:08:30.440159 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:08:30.839358 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:30.839290 4455 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 18:08:30.839623 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:30.839598 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" podUID="ac80ccea-a611-4de1-951c-479ad3d827a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 18:08:31.114852 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:31.114750 4455 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 18:08:31.115136 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:31.115104 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" podUID="d7de539f-17e9-453c-af6b-e83fc7c91239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 18:08:31.792349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:31.791478 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv"] Oct 30 18:08:31.815848 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:31.814869 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" Oct 30 18:08:31.816423 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:31.816047 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv"] Oct 30 18:08:31.839197 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:31.839164 4455 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 18:08:31.839397 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:31.839215 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" podUID="ac80ccea-a611-4de1-951c-479ad3d827a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 18:08:31.955254 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:31.955217 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b933cb99-2e87-4707-a33b-de09b5c03536-istio-data\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" Oct 30 18:08:31.955516 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:31.955260 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnfvq\" (UniqueName: \"kubernetes.io/projected/b933cb99-2e87-4707-a33b-de09b5c03536-kube-api-access-rnfvq\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" Oct 30 18:08:31.955516 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:31.955293 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b933cb99-2e87-4707-a33b-de09b5c03536-credential-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" Oct 30 18:08:31.955516 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:31.955448 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b933cb99-2e87-4707-a33b-de09b5c03536-istio-envoy\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" Oct 30 18:08:31.955703 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:31.955518 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b933cb99-2e87-4707-a33b-de09b5c03536-istio-token\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" Oct 30 18:08:31.955703 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:31.955550 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b933cb99-2e87-4707-a33b-de09b5c03536-istiod-ca-cert\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" Oct 30 18:08:31.955703 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:31.955674 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b933cb99-2e87-4707-a33b-de09b5c03536-istio-podinfo\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" Oct 30 18:08:31.955857 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:31.955837 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b933cb99-2e87-4707-a33b-de09b5c03536-workload-certs\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" Oct 30 18:08:31.956014 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:31.955989 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b933cb99-2e87-4707-a33b-de09b5c03536-workload-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" Oct 30 18:08:32.057121 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:32.057031 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b933cb99-2e87-4707-a33b-de09b5c03536-istio-podinfo\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" Oct 30 18:08:32.057287 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:32.057165 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b933cb99-2e87-4707-a33b-de09b5c03536-workload-certs\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" Oct 30 18:08:32.057287 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:32.057271 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b933cb99-2e87-4707-a33b-de09b5c03536-workload-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" Oct 30 18:08:32.057443 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:32.057372 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b933cb99-2e87-4707-a33b-de09b5c03536-istio-data\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" Oct 30 18:08:32.057443 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:32.057399 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnfvq\" (UniqueName: \"kubernetes.io/projected/b933cb99-2e87-4707-a33b-de09b5c03536-kube-api-access-rnfvq\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" Oct 30 18:08:32.057443 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:32.057427 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b933cb99-2e87-4707-a33b-de09b5c03536-credential-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" Oct 30 18:08:32.057611 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:32.057477 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b933cb99-2e87-4707-a33b-de09b5c03536-istio-envoy\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" Oct 30 18:08:32.057611 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:32.057506 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b933cb99-2e87-4707-a33b-de09b5c03536-istio-token\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" Oct 30 18:08:32.057611 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:32.057531 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b933cb99-2e87-4707-a33b-de09b5c03536-istiod-ca-cert\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" Oct 30 18:08:32.057770 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:32.057682 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b933cb99-2e87-4707-a33b-de09b5c03536-workload-certs\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" Oct 30 18:08:32.058124 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:32.058099 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b933cb99-2e87-4707-a33b-de09b5c03536-workload-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" Oct 30 18:08:32.058362 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:32.058306 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b933cb99-2e87-4707-a33b-de09b5c03536-istio-data\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" Oct 30 18:08:32.058466 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:32.058312 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b933cb99-2e87-4707-a33b-de09b5c03536-istiod-ca-cert\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" Oct 30 18:08:32.058787 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:32.058757 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b933cb99-2e87-4707-a33b-de09b5c03536-credential-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" Oct 30 18:08:32.060479 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:32.060451 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b933cb99-2e87-4707-a33b-de09b5c03536-istio-envoy\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" Oct 30 18:08:32.060788 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:32.060767 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b933cb99-2e87-4707-a33b-de09b5c03536-istio-podinfo\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" Oct 30 18:08:32.082843 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:32.082813 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b933cb99-2e87-4707-a33b-de09b5c03536-istio-token\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" Oct 30 18:08:32.083515 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:32.083495 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnfvq\" (UniqueName: \"kubernetes.io/projected/b933cb99-2e87-4707-a33b-de09b5c03536-kube-api-access-rnfvq\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" Oct 30 18:08:32.113483 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:32.113445 4455 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 18:08:32.113672 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:32.113504 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" podUID="d7de539f-17e9-453c-af6b-e83fc7c91239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 18:08:32.145708 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:32.145675 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" Oct 30 18:08:32.506021 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:32.505961 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:08:32.506154 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:32.506124 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:08:32.506274 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:32.506245 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:08:32.506903 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:32.506876 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv"] Oct 30 18:08:32.834137 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:32.834019 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" event={"ID":"b933cb99-2e87-4707-a33b-de09b5c03536","Type":"ContainerStarted","Data":"94025eff33042a330f779ddb82ed265e6b2000e1cebc1a543c190af822e14a08"} Oct 30 18:08:32.834137 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:32.834092 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" event={"ID":"b933cb99-2e87-4707-a33b-de09b5c03536","Type":"ContainerStarted","Data":"125fa24784c534b6f3dbcb38ea37748d945cbb0bfacea8203f2912e79eaffb61"} Oct 30 18:08:32.931070 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:32.931039 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" Oct 30 18:08:32.931535 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:32.931505 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" Oct 30 18:08:32.933972 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:32.933214 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" Oct 30 18:08:32.981540 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:32.981491 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" podStartSLOduration=1.9814748610000001 podStartE2EDuration="1.981474861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:08:32.899619666 +0000 UTC m=+965.387256709" watchObservedRunningTime="2026-10-30 18:08:32.981474861 +0000 UTC m=+965.469111892" Oct 30 18:08:33.151440 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:33.149801 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" Oct 30 18:08:33.151440 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:33.149942 4455 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 18:08:33.151440 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:33.149988 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" podUID="b933cb99-2e87-4707-a33b-de09b5c03536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 18:08:33.717903 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:33.717871 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" Oct 30 18:08:33.718648 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:33.718587 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" Oct 30 18:08:33.719809 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:33.719781 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" Oct 30 18:08:34.146470 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:34.146384 4455 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 18:08:34.146470 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:34.146438 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" podUID="b933cb99-2e87-4707-a33b-de09b5c03536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 18:08:35.146675 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:35.146639 4455 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 18:08:35.147058 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:35.146711 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" podUID="b933cb99-2e87-4707-a33b-de09b5c03536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 18:08:36.146129 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:36.146095 4455 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 18:08:36.146310 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:36.146145 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" podUID="b933cb99-2e87-4707-a33b-de09b5c03536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 18:08:37.147373 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:37.146113 4455 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 18:08:37.147373 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:37.146177 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" podUID="b933cb99-2e87-4707-a33b-de09b5c03536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 18:08:38.146556 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:38.146522 4455 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 18:08:38.146742 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:38.146586 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" podUID="b933cb99-2e87-4707-a33b-de09b5c03536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 18:08:39.146080 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:39.146030 4455 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 18:08:39.146651 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:39.146090 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" podUID="b933cb99-2e87-4707-a33b-de09b5c03536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 18:08:39.797599 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:39.797558 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg"] Oct 30 18:08:39.802649 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:39.802624 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" Oct 30 18:08:39.830113 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:39.830080 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg"] Oct 30 18:08:39.860359 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:39.860293 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70d7e6c8-8d0f-40f0-a533-58158542fb7c-istiod-ca-cert\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" Oct 30 18:08:39.860535 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:39.860440 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d7e6c8-8d0f-40f0-a533-58158542fb7c-credential-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" Oct 30 18:08:39.860535 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:39.860480 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92vhn\" (UniqueName: \"kubernetes.io/projected/70d7e6c8-8d0f-40f0-a533-58158542fb7c-kube-api-access-92vhn\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" Oct 30 18:08:39.860535 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:39.860503 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70d7e6c8-8d0f-40f0-a533-58158542fb7c-istio-envoy\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" Oct 30 18:08:39.860740 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:39.860589 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70d7e6c8-8d0f-40f0-a533-58158542fb7c-istio-data\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" Oct 30 18:08:39.860740 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:39.860663 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70d7e6c8-8d0f-40f0-a533-58158542fb7c-istio-podinfo\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" Oct 30 18:08:39.860855 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:39.860809 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d7e6c8-8d0f-40f0-a533-58158542fb7c-workload-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" Oct 30 18:08:39.860937 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:39.860915 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70d7e6c8-8d0f-40f0-a533-58158542fb7c-workload-certs\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" Oct 30 18:08:39.861001 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:39.860968 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70d7e6c8-8d0f-40f0-a533-58158542fb7c-istio-token\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" Oct 30 18:08:39.962091 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:39.962052 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70d7e6c8-8d0f-40f0-a533-58158542fb7c-istiod-ca-cert\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" Oct 30 18:08:39.962301 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:39.962155 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d7e6c8-8d0f-40f0-a533-58158542fb7c-credential-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" Oct 30 18:08:39.962301 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:39.962190 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92vhn\" (UniqueName: \"kubernetes.io/projected/70d7e6c8-8d0f-40f0-a533-58158542fb7c-kube-api-access-92vhn\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" Oct 30 18:08:39.962301 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:39.962227 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70d7e6c8-8d0f-40f0-a533-58158542fb7c-istio-envoy\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" Oct 30 18:08:39.962301 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:39.962273 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70d7e6c8-8d0f-40f0-a533-58158542fb7c-istio-data\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" Oct 30 18:08:39.962525 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:39.962312 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70d7e6c8-8d0f-40f0-a533-58158542fb7c-istio-podinfo\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" Oct 30 18:08:39.962525 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:39.962451 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d7e6c8-8d0f-40f0-a533-58158542fb7c-workload-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" Oct 30 18:08:39.962632 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:39.962551 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70d7e6c8-8d0f-40f0-a533-58158542fb7c-workload-certs\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" Oct 30 18:08:39.962632 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:39.962599 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70d7e6c8-8d0f-40f0-a533-58158542fb7c-istio-token\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" Oct 30 18:08:39.962832 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:39.962787 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70d7e6c8-8d0f-40f0-a533-58158542fb7c-istio-data\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" Oct 30 18:08:39.962900 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:39.962812 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d7e6c8-8d0f-40f0-a533-58158542fb7c-credential-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" Oct 30 18:08:39.962957 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:39.962892 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70d7e6c8-8d0f-40f0-a533-58158542fb7c-istiod-ca-cert\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" Oct 30 18:08:39.963011 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:39.962964 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d7e6c8-8d0f-40f0-a533-58158542fb7c-workload-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" Oct 30 18:08:39.963332 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:39.963299 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70d7e6c8-8d0f-40f0-a533-58158542fb7c-workload-certs\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" Oct 30 18:08:39.964953 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:39.964933 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70d7e6c8-8d0f-40f0-a533-58158542fb7c-istio-podinfo\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" Oct 30 18:08:39.966075 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:39.966057 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70d7e6c8-8d0f-40f0-a533-58158542fb7c-istio-envoy\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" Oct 30 18:08:39.992430 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:39.992392 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70d7e6c8-8d0f-40f0-a533-58158542fb7c-istio-token\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" Oct 30 18:08:39.992589 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:39.992456 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92vhn\" (UniqueName: \"kubernetes.io/projected/70d7e6c8-8d0f-40f0-a533-58158542fb7c-kube-api-access-92vhn\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" Oct 30 18:08:40.121489 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:40.121390 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" Oct 30 18:08:40.150367 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:40.150305 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" Oct 30 18:08:40.150771 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:40.150646 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" Oct 30 18:08:40.151489 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:40.151471 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" Oct 30 18:08:40.445295 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:08:40.445264 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70d7e6c8_8d0f_40f0_a533_58158542fb7c.slice/crio-d4405654400a6047c17679af6e61c2bcc5cd8b0a8bb8daec54431afd53fbba4a WatchSource:0}: Error finding container d4405654400a6047c17679af6e61c2bcc5cd8b0a8bb8daec54431afd53fbba4a: Status 404 returned error can't find the container with id d4405654400a6047c17679af6e61c2bcc5cd8b0a8bb8daec54431afd53fbba4a Oct 30 18:08:40.448094 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:40.448059 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:08:40.448175 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:40.448129 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:08:40.448175 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:40.448163 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:08:40.448936 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:40.448879 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg"] Oct 30 18:08:40.953453 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:40.953414 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" event={"ID":"70d7e6c8-8d0f-40f0-a533-58158542fb7c","Type":"ContainerStarted","Data":"80af84e433f2921c01f3cb8069dc5888444f4e02f28294a6f2f627170a368f6f"} Oct 30 18:08:40.953699 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:40.953484 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" event={"ID":"70d7e6c8-8d0f-40f0-a533-58158542fb7c","Type":"ContainerStarted","Data":"d4405654400a6047c17679af6e61c2bcc5cd8b0a8bb8daec54431afd53fbba4a"} Oct 30 18:08:41.005194 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:41.005133 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" podStartSLOduration=2.005118006 podStartE2EDuration="2.005118006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:08:41.002150726 +0000 UTC m=+973.489787761" watchObservedRunningTime="2026-10-30 18:08:41.005118006 +0000 UTC m=+973.492755107" Oct 30 18:08:41.122581 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:41.122539 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" Oct 30 18:08:41.124504 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:41.124481 4455 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 18:08:41.124602 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:41.124543 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" podUID="70d7e6c8-8d0f-40f0-a533-58158542fb7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 18:08:42.122349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:42.122290 4455 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 18:08:42.122775 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:42.122397 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" podUID="70d7e6c8-8d0f-40f0-a533-58158542fb7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 18:08:43.121898 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:43.121843 4455 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 18:08:43.122072 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:43.121907 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" podUID="70d7e6c8-8d0f-40f0-a533-58158542fb7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 18:08:44.122764 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:44.122726 4455 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 18:08:44.125185 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:44.122781 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" podUID="70d7e6c8-8d0f-40f0-a533-58158542fb7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 18:08:45.122709 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:45.122670 4455 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 18:08:45.122929 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:45.122724 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" podUID="70d7e6c8-8d0f-40f0-a533-58158542fb7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 18:08:46.122383 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:46.122297 4455 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 18:08:46.122569 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:46.122412 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" podUID="70d7e6c8-8d0f-40f0-a533-58158542fb7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 18:08:47.122470 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:47.122435 4455 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 18:08:47.122867 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:47.122492 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" podUID="70d7e6c8-8d0f-40f0-a533-58158542fb7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 18:08:48.173279 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:48.173042 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" Oct 30 18:08:48.174019 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:48.173603 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" Oct 30 18:08:48.174380 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:48.174315 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" Oct 30 18:08:58.693995 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:58.693465 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:08:58.693995 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:58.693551 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" podUID="c7c21be6-b9eb-401c-aa3c-af5b62960cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:58.995528 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:58.995486 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:08:58.996294 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:08:58.995861 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" podUID="4e7b0c73-0041-4549-b47e-76a7aaed2b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:00.861675 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:09:00.861581 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:09:09.705631 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:09.705589 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26"] Oct 30 18:09:09.712477 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:09.712451 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" Oct 30 18:09:09.734813 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:09.733773 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26"] Oct 30 18:09:09.817701 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:09.817670 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b710b6-b5a8-4fe0-ac70-861dd036b70d-istio-data\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" Oct 30 18:09:09.817896 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:09.817710 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gjld\" (UniqueName: \"kubernetes.io/projected/93b710b6-b5a8-4fe0-ac70-861dd036b70d-kube-api-access-6gjld\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" Oct 30 18:09:09.817896 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:09.817750 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b710b6-b5a8-4fe0-ac70-861dd036b70d-istio-token\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" Oct 30 18:09:09.817896 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:09.817846 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b710b6-b5a8-4fe0-ac70-861dd036b70d-credential-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" Oct 30 18:09:09.817896 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:09.817879 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b710b6-b5a8-4fe0-ac70-861dd036b70d-istio-envoy\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" Oct 30 18:09:09.818056 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:09.817909 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b710b6-b5a8-4fe0-ac70-861dd036b70d-workload-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" Oct 30 18:09:09.818056 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:09.817980 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b710b6-b5a8-4fe0-ac70-861dd036b70d-workload-certs\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" Oct 30 18:09:09.818130 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:09.818058 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b710b6-b5a8-4fe0-ac70-861dd036b70d-istio-podinfo\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" Oct 30 18:09:09.818130 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:09.818091 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b710b6-b5a8-4fe0-ac70-861dd036b70d-istiod-ca-cert\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" Oct 30 18:09:09.919548 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:09.919514 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b710b6-b5a8-4fe0-ac70-861dd036b70d-istio-data\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" Oct 30 18:09:09.919720 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:09.919555 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gjld\" (UniqueName: \"kubernetes.io/projected/93b710b6-b5a8-4fe0-ac70-861dd036b70d-kube-api-access-6gjld\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" Oct 30 18:09:09.919720 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:09.919599 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b710b6-b5a8-4fe0-ac70-861dd036b70d-istio-token\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" Oct 30 18:09:09.919720 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:09.919630 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b710b6-b5a8-4fe0-ac70-861dd036b70d-credential-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" Oct 30 18:09:09.919720 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:09.919651 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b710b6-b5a8-4fe0-ac70-861dd036b70d-istio-envoy\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" Oct 30 18:09:09.919720 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:09.919691 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b710b6-b5a8-4fe0-ac70-861dd036b70d-workload-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" Oct 30 18:09:09.919993 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:09.919759 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b710b6-b5a8-4fe0-ac70-861dd036b70d-workload-certs\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" Oct 30 18:09:09.919993 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:09.919824 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b710b6-b5a8-4fe0-ac70-861dd036b70d-istio-podinfo\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" Oct 30 18:09:09.919993 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:09.919852 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b710b6-b5a8-4fe0-ac70-861dd036b70d-istiod-ca-cert\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" Oct 30 18:09:09.920141 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:09.920069 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b710b6-b5a8-4fe0-ac70-861dd036b70d-credential-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" Oct 30 18:09:09.920141 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:09.920131 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b710b6-b5a8-4fe0-ac70-861dd036b70d-workload-certs\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" Oct 30 18:09:09.920212 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:09.920169 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b710b6-b5a8-4fe0-ac70-861dd036b70d-workload-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" Oct 30 18:09:09.920307 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:09.920288 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b710b6-b5a8-4fe0-ac70-861dd036b70d-istio-data\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" Oct 30 18:09:09.920652 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:09.920635 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b710b6-b5a8-4fe0-ac70-861dd036b70d-istiod-ca-cert\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" Oct 30 18:09:09.922011 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:09.921990 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b710b6-b5a8-4fe0-ac70-861dd036b70d-istio-envoy\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" Oct 30 18:09:09.922430 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:09.922409 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b710b6-b5a8-4fe0-ac70-861dd036b70d-istio-podinfo\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" Oct 30 18:09:09.944787 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:09.944753 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b710b6-b5a8-4fe0-ac70-861dd036b70d-istio-token\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" Oct 30 18:09:09.945061 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:09.945039 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gjld\" (UniqueName: \"kubernetes.io/projected/93b710b6-b5a8-4fe0-ac70-861dd036b70d-kube-api-access-6gjld\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" Oct 30 18:09:10.043735 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:10.043697 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" Oct 30 18:09:10.361293 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:09:10.361261 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b710b6_b5a8_4fe0_ac70_861dd036b70d.slice/crio-aaa60f6ed92f62b1ac953df682edf79e8151a786a420e01cac0cfe72ee1a9d41 WatchSource:0}: Error finding container aaa60f6ed92f62b1ac953df682edf79e8151a786a420e01cac0cfe72ee1a9d41: Status 404 returned error can't find the container with id aaa60f6ed92f62b1ac953df682edf79e8151a786a420e01cac0cfe72ee1a9d41 Oct 30 18:09:10.362871 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:10.362853 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26"] Oct 30 18:09:10.363621 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:10.363590 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:09:10.363667 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:10.363655 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:09:10.363701 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:10.363684 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:09:10.978601 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:10.978549 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" event={"ID":"93b710b6-b5a8-4fe0-ac70-861dd036b70d","Type":"ContainerStarted","Data":"b693433aac4f98920aa85577a8ea16b04a62c0737ed7789f243a93630cc5955d"} Oct 30 18:09:10.979435 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:10.979406 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" event={"ID":"93b710b6-b5a8-4fe0-ac70-861dd036b70d","Type":"ContainerStarted","Data":"aaa60f6ed92f62b1ac953df682edf79e8151a786a420e01cac0cfe72ee1a9d41"} Oct 30 18:09:11.036054 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:11.036000 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" podStartSLOduration=2.035982802 podStartE2EDuration="2.035982802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:11.03254936 +0000 UTC m=+1003.520186420" watchObservedRunningTime="2026-10-30 18:09:11.035982802 +0000 UTC m=+1003.523619837" Oct 30 18:09:11.044023 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:11.043992 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" Oct 30 18:09:11.045730 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:11.045703 4455 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 18:09:11.045823 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:11.045763 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" podUID="93b710b6-b5a8-4fe0-ac70-861dd036b70d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 18:09:12.044925 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:12.044888 4455 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 18:09:12.045359 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:12.044942 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" podUID="93b710b6-b5a8-4fe0-ac70-861dd036b70d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 18:09:13.045005 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:13.044968 4455 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 18:09:13.045444 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:13.045025 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" podUID="93b710b6-b5a8-4fe0-ac70-861dd036b70d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 18:09:14.044813 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:14.044783 4455 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 18:09:14.044996 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:14.044832 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" podUID="93b710b6-b5a8-4fe0-ac70-861dd036b70d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 18:09:15.045018 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:15.044981 4455 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 18:09:15.045465 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:15.045034 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" podUID="93b710b6-b5a8-4fe0-ac70-861dd036b70d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 18:09:16.044854 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:16.044820 4455 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 18:09:16.045031 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:16.044872 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" podUID="93b710b6-b5a8-4fe0-ac70-861dd036b70d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 18:09:17.045075 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:17.045031 4455 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 18:09:17.045569 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:17.045097 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" podUID="93b710b6-b5a8-4fe0-ac70-861dd036b70d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 18:09:18.083586 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:18.083554 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" Oct 30 18:09:18.084910 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:18.084880 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" Oct 30 18:09:18.085826 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:18.085804 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" Oct 30 18:09:30.387297 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:09:30.387258 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:09:35.998438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:35.998402 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb"] Oct 30 18:09:36.004534 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:36.004508 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:09:36.030385 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:36.028886 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb"] Oct 30 18:09:36.036658 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:36.036632 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-6e8fca\"/\"kube-root-ca.crt\"" Oct 30 18:09:36.036847 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:36.036632 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-6e8fca\"/\"openshift-service-ca.crt\"" Oct 30 18:09:36.036941 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:36.036688 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-6e8fca\"/\"distinct-2-w2-6e8fca-kserve-self-signed-certs\"" Oct 30 18:09:36.036941 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:36.036749 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-6e8fca\"/\"default-dockercfg-d6n2t\"" Oct 30 18:09:36.077251 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:36.077217 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-home\") pod \"distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb\" (UID: \"d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:09:36.077251 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:36.077256 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-tls-certs\") pod \"distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb\" (UID: \"d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:09:36.077487 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:36.077425 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-model-cache\") pod \"distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb\" (UID: \"d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:09:36.077487 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:36.077481 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-dshm\") pod \"distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb\" (UID: \"d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:09:36.077602 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:36.077584 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtstq\" (UniqueName: \"kubernetes.io/projected/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-kube-api-access-xtstq\") pod \"distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb\" (UID: \"d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:09:36.077638 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:36.077623 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-tmp-dir\") pod \"distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb\" (UID: \"d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:09:36.178516 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:36.178482 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-dshm\") pod \"distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb\" (UID: \"d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:09:36.178688 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:36.178581 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtstq\" (UniqueName: \"kubernetes.io/projected/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-kube-api-access-xtstq\") pod \"distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb\" (UID: \"d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:09:36.178688 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:36.178624 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-tmp-dir\") pod \"distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb\" (UID: \"d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:09:36.178688 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:36.178665 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-home\") pod \"distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb\" (UID: \"d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:09:36.178854 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:36.178695 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-tls-certs\") pod \"distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb\" (UID: \"d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:09:36.178854 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:36.178837 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-model-cache\") pod \"distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb\" (UID: \"d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:09:36.179094 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:36.179065 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-tmp-dir\") pod \"distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb\" (UID: \"d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:09:36.179175 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:36.179157 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-home\") pod \"distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb\" (UID: \"d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:09:36.179308 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:36.179252 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-model-cache\") pod \"distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb\" (UID: \"d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:09:36.181006 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:36.180983 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-dshm\") pod \"distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb\" (UID: \"d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:09:36.181504 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:36.181487 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-tls-certs\") pod \"distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb\" (UID: \"d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:09:36.226164 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:36.226054 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtstq\" (UniqueName: \"kubernetes.io/projected/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-kube-api-access-xtstq\") pod \"distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb\" (UID: \"d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:09:36.333362 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:36.333247 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:09:36.679177 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:09:36.679144 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3ad09e5_ba36_401c_b93b_32bcd2f6c5c7.slice/crio-96b20b1cc872f6fad763dd32712cd93e850fad51ca70375a390f51d216d0660d WatchSource:0}: Error finding container 96b20b1cc872f6fad763dd32712cd93e850fad51ca70375a390f51d216d0660d: Status 404 returned error can't find the container with id 96b20b1cc872f6fad763dd32712cd93e850fad51ca70375a390f51d216d0660d Oct 30 18:09:36.680616 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:36.680592 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb"] Oct 30 18:09:37.198665 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:37.198627 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" event={"ID":"d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7","Type":"ContainerStarted","Data":"144822bba13c775f3ce8d2f977524f702498a6d4cf18a945bf8adb693cacafc8"} Oct 30 18:09:37.198665 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:37.198671 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" event={"ID":"d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7","Type":"ContainerStarted","Data":"96b20b1cc872f6fad763dd32712cd93e850fad51ca70375a390f51d216d0660d"} Oct 30 18:09:37.244226 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:37.244174 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podStartSLOduration=2.244161654 podStartE2EDuration="2.244161654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:37.243149096 +0000 UTC m=+1029.730786132" watchObservedRunningTime="2026-10-30 18:09:37.244161654 +0000 UTC m=+1029.731798688" Oct 30 18:09:37.334177 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:37.334140 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:09:46.352464 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:46.338812 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:09:46.437352 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:46.431978 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:09:47.931359 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:09:47.923968 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:10:00.386646 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:10:00.386599 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:10:22.129228 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:22.129190 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6c5884dffd-xh5zr"] Oct 30 18:10:22.133868 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:22.133849 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c5884dffd-xh5zr" Oct 30 18:10:22.155288 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:22.155256 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c5884dffd-xh5zr"] Oct 30 18:10:22.265109 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:22.265065 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9554714f-5b63-4cec-80a1-e154f0d48142-routing-overlay-volume\") pod \"payload-processing-external-model-6c5884dffd-xh5zr\" (UID: \"9554714f-5b63-4cec-80a1-e154f0d48142\") " pod="models-as-a-service/payload-processing-external-model-6c5884dffd-xh5zr" Oct 30 18:10:22.265388 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:22.265367 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9554714f-5b63-4cec-80a1-e154f0d48142-plugins-config-volume\") pod \"payload-processing-external-model-6c5884dffd-xh5zr\" (UID: \"9554714f-5b63-4cec-80a1-e154f0d48142\") " pod="models-as-a-service/payload-processing-external-model-6c5884dffd-xh5zr" Oct 30 18:10:22.265474 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:22.265460 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9554714f-5b63-4cec-80a1-e154f0d48142-provider-credentials\") pod \"payload-processing-external-model-6c5884dffd-xh5zr\" (UID: \"9554714f-5b63-4cec-80a1-e154f0d48142\") " pod="models-as-a-service/payload-processing-external-model-6c5884dffd-xh5zr" Oct 30 18:10:22.265559 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:22.265538 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5s9r\" (UniqueName: \"kubernetes.io/projected/9554714f-5b63-4cec-80a1-e154f0d48142-kube-api-access-l5s9r\") pod \"payload-processing-external-model-6c5884dffd-xh5zr\" (UID: \"9554714f-5b63-4cec-80a1-e154f0d48142\") " pod="models-as-a-service/payload-processing-external-model-6c5884dffd-xh5zr" Oct 30 18:10:22.366271 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:22.366235 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9554714f-5b63-4cec-80a1-e154f0d48142-plugins-config-volume\") pod \"payload-processing-external-model-6c5884dffd-xh5zr\" (UID: \"9554714f-5b63-4cec-80a1-e154f0d48142\") " pod="models-as-a-service/payload-processing-external-model-6c5884dffd-xh5zr" Oct 30 18:10:22.366509 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:22.366311 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9554714f-5b63-4cec-80a1-e154f0d48142-provider-credentials\") pod \"payload-processing-external-model-6c5884dffd-xh5zr\" (UID: \"9554714f-5b63-4cec-80a1-e154f0d48142\") " pod="models-as-a-service/payload-processing-external-model-6c5884dffd-xh5zr" Oct 30 18:10:22.366509 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:22.366382 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5s9r\" (UniqueName: \"kubernetes.io/projected/9554714f-5b63-4cec-80a1-e154f0d48142-kube-api-access-l5s9r\") pod \"payload-processing-external-model-6c5884dffd-xh5zr\" (UID: \"9554714f-5b63-4cec-80a1-e154f0d48142\") " pod="models-as-a-service/payload-processing-external-model-6c5884dffd-xh5zr" Oct 30 18:10:22.366509 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:22.366415 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9554714f-5b63-4cec-80a1-e154f0d48142-routing-overlay-volume\") pod \"payload-processing-external-model-6c5884dffd-xh5zr\" (UID: \"9554714f-5b63-4cec-80a1-e154f0d48142\") " pod="models-as-a-service/payload-processing-external-model-6c5884dffd-xh5zr" Oct 30 18:10:22.367163 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:22.367137 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9554714f-5b63-4cec-80a1-e154f0d48142-plugins-config-volume\") pod \"payload-processing-external-model-6c5884dffd-xh5zr\" (UID: \"9554714f-5b63-4cec-80a1-e154f0d48142\") " pod="models-as-a-service/payload-processing-external-model-6c5884dffd-xh5zr" Oct 30 18:10:22.367258 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:22.367174 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9554714f-5b63-4cec-80a1-e154f0d48142-routing-overlay-volume\") pod \"payload-processing-external-model-6c5884dffd-xh5zr\" (UID: \"9554714f-5b63-4cec-80a1-e154f0d48142\") " pod="models-as-a-service/payload-processing-external-model-6c5884dffd-xh5zr" Oct 30 18:10:22.369040 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:22.369017 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9554714f-5b63-4cec-80a1-e154f0d48142-provider-credentials\") pod \"payload-processing-external-model-6c5884dffd-xh5zr\" (UID: \"9554714f-5b63-4cec-80a1-e154f0d48142\") " pod="models-as-a-service/payload-processing-external-model-6c5884dffd-xh5zr" Oct 30 18:10:22.391493 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:22.391430 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5s9r\" (UniqueName: \"kubernetes.io/projected/9554714f-5b63-4cec-80a1-e154f0d48142-kube-api-access-l5s9r\") pod \"payload-processing-external-model-6c5884dffd-xh5zr\" (UID: \"9554714f-5b63-4cec-80a1-e154f0d48142\") " pod="models-as-a-service/payload-processing-external-model-6c5884dffd-xh5zr" Oct 30 18:10:22.458067 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:22.458030 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c5884dffd-xh5zr" Oct 30 18:10:23.020817 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:10:23.020783 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9554714f_5b63_4cec_80a1_e154f0d48142.slice/crio-931bac847fe9c4188ff0950f56db249700398f250de9af5c4858c5a92176ade4 WatchSource:0}: Error finding container 931bac847fe9c4188ff0950f56db249700398f250de9af5c4858c5a92176ade4: Status 404 returned error can't find the container with id 931bac847fe9c4188ff0950f56db249700398f250de9af5c4858c5a92176ade4 Oct 30 18:10:23.022573 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:23.022546 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c5884dffd-xh5zr"] Oct 30 18:10:23.899619 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:23.899581 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c5884dffd-xh5zr" event={"ID":"9554714f-5b63-4cec-80a1-e154f0d48142","Type":"ContainerStarted","Data":"3dfe793828830fb90b8c533aa22471bf75a3a372a896194ae106504c0fa326cb"} Oct 30 18:10:23.899994 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:23.899642 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c5884dffd-xh5zr" event={"ID":"9554714f-5b63-4cec-80a1-e154f0d48142","Type":"ContainerStarted","Data":"931bac847fe9c4188ff0950f56db249700398f250de9af5c4858c5a92176ade4"} Oct 30 18:10:23.899994 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:23.899670 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6c5884dffd-xh5zr" Oct 30 18:10:23.945801 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:23.945745 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6c5884dffd-xh5zr" podStartSLOduration=1.9457318300000002 podStartE2EDuration="1.94573183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:23.943465922 +0000 UTC m=+1076.431102979" watchObservedRunningTime="2026-10-30 18:10:23.94573183 +0000 UTC m=+1076.433368863" Oct 30 18:10:30.455597 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:10:30.455536 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:10:33.095466 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:33.095426 4455 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:10:33.096088 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:33.096054 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" podUID="d4e8e077-c9f7-44c8-9dd5-d1c7a2403862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:34.356590 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:34.356550 4455 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:10:34.357203 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:34.357167 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" podUID="93b710b6-b5a8-4fe0-ac70-861dd036b70d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:34.357734 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:34.357706 4455 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:10:34.357924 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:34.357900 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" podUID="70d7e6c8-8d0f-40f0-a533-58158542fb7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:34.961351 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:34.945906 4455 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:10:34.961351 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:34.946018 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" podUID="d7de539f-17e9-453c-af6b-e83fc7c91239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:35.121370 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:35.112514 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6c5884dffd-xh5zr" Oct 30 18:10:35.460154 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:35.444668 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-78d8ccc49c-8txzp"] Oct 30 18:10:38.088827 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:38.088441 4455 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:10:38.088827 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:38.088537 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" podUID="bbf4bc50-3670-40de-8159-3d8805fdd601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:38.329376 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:38.328974 4455 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:10:38.329376 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:38.329048 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" podUID="3daa8589-9def-486f-82ee-0c923a18b68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:40.463368 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:40.461230 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-78d8ccc49c-8txzp" podUID="f0b1e9d0-b896-4a83-844d-d566d3d324d6" containerName="payload-processing" containerID="cri-o://046b6dd409fb81acd39cd06f84189bbf263e7e0165243c376f25005134f786ec" gracePeriod=25 Oct 30 18:10:40.768213 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:40.768184 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78d8ccc49c-8txzp" Oct 30 18:10:40.868411 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:40.868370 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0b1e9d0-b896-4a83-844d-d566d3d324d6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0b1e9d0-b896-4a83-844d-d566d3d324d6-routing-overlay-volume\") pod \"f0b1e9d0-b896-4a83-844d-d566d3d324d6\" (UID: \"f0b1e9d0-b896-4a83-844d-d566d3d324d6\") " Oct 30 18:10:40.868609 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:40.868452 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0b1e9d0-b896-4a83-844d-d566d3d324d6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0b1e9d0-b896-4a83-844d-d566d3d324d6-provider-credentials\") pod \"f0b1e9d0-b896-4a83-844d-d566d3d324d6\" (UID: \"f0b1e9d0-b896-4a83-844d-d566d3d324d6\") " Oct 30 18:10:40.868609 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:40.868478 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0b1e9d0-b896-4a83-844d-d566d3d324d6-kube-api-access-r6vqw\" (UniqueName: \"kubernetes.io/projected/f0b1e9d0-b896-4a83-844d-d566d3d324d6-kube-api-access-r6vqw\") pod \"f0b1e9d0-b896-4a83-844d-d566d3d324d6\" (UID: \"f0b1e9d0-b896-4a83-844d-d566d3d324d6\") " Oct 30 18:10:40.868609 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:40.868544 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0b1e9d0-b896-4a83-844d-d566d3d324d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0b1e9d0-b896-4a83-844d-d566d3d324d6-plugins-config-volume\") pod \"f0b1e9d0-b896-4a83-844d-d566d3d324d6\" (UID: \"f0b1e9d0-b896-4a83-844d-d566d3d324d6\") " Oct 30 18:10:40.868823 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:40.868793 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0b1e9d0-b896-4a83-844d-d566d3d324d6-routing-overlay-volume" pod "f0b1e9d0-b896-4a83-844d-d566d3d324d6" (UID: "f0b1e9d0-b896-4a83-844d-d566d3d324d6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:10:40.869058 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:40.869020 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0b1e9d0-b896-4a83-844d-d566d3d324d6-plugins-config-volume" pod "f0b1e9d0-b896-4a83-844d-d566d3d324d6" (UID: "f0b1e9d0-b896-4a83-844d-d566d3d324d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:10:40.869591 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:40.869567 4455 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0b1e9d0-b896-4a83-844d-d566d3d324d6-routing-overlay-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:10:40.869691 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:40.869596 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0b1e9d0-b896-4a83-844d-d566d3d324d6-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:10:40.871049 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:40.871019 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0b1e9d0-b896-4a83-844d-d566d3d324d6-kube-api-access-r6vqw" pod "f0b1e9d0-b896-4a83-844d-d566d3d324d6" (UID: "f0b1e9d0-b896-4a83-844d-d566d3d324d6"). InnerVolumeSpecName "kube-api-access-r6vqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:10:40.871383 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:40.871354 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0b1e9d0-b896-4a83-844d-d566d3d324d6-provider-credentials" pod "f0b1e9d0-b896-4a83-844d-d566d3d324d6" (UID: "f0b1e9d0-b896-4a83-844d-d566d3d324d6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:10:40.970904 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:40.970858 4455 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0b1e9d0-b896-4a83-844d-d566d3d324d6-provider-credentials\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:10:40.970904 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:40.970889 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6vqw\" (UniqueName: \"kubernetes.io/projected/f0b1e9d0-b896-4a83-844d-d566d3d324d6-kube-api-access-r6vqw\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:10:41.453862 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:41.453824 4455 generic.go:356] "Generic (PLEG): container finished" podID="f0b1e9d0-b896-4a83-844d-d566d3d324d6" containerID="046b6dd409fb81acd39cd06f84189bbf263e7e0165243c376f25005134f786ec" exitCode=0 Oct 30 18:10:41.454079 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:41.453868 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78d8ccc49c-8txzp" Oct 30 18:10:41.454079 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:41.453914 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78d8ccc49c-8txzp" event={"ID":"f0b1e9d0-b896-4a83-844d-d566d3d324d6","Type":"ContainerDied","Data":"046b6dd409fb81acd39cd06f84189bbf263e7e0165243c376f25005134f786ec"} Oct 30 18:10:41.454079 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:41.453974 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78d8ccc49c-8txzp" event={"ID":"f0b1e9d0-b896-4a83-844d-d566d3d324d6","Type":"ContainerDied","Data":"6a059a03a2f3f09c579f3eabe457d73e7f3bd36bbed1117d0312d13975c14883"} Oct 30 18:10:41.454079 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:41.454003 4455 scope.go:122] "RemoveContainer" containerID="046b6dd409fb81acd39cd06f84189bbf263e7e0165243c376f25005134f786ec" Oct 30 18:10:41.465862 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:41.465666 4455 scope.go:122] "RemoveContainer" containerID="046b6dd409fb81acd39cd06f84189bbf263e7e0165243c376f25005134f786ec" Oct 30 18:10:41.466134 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:10:41.465990 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"046b6dd409fb81acd39cd06f84189bbf263e7e0165243c376f25005134f786ec\": container with ID starting with 046b6dd409fb81acd39cd06f84189bbf263e7e0165243c376f25005134f786ec not found: ID does not exist" containerID="046b6dd409fb81acd39cd06f84189bbf263e7e0165243c376f25005134f786ec" Oct 30 18:10:41.466134 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:41.466018 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"046b6dd409fb81acd39cd06f84189bbf263e7e0165243c376f25005134f786ec"} err="failed to get container status \"046b6dd409fb81acd39cd06f84189bbf263e7e0165243c376f25005134f786ec\": rpc error: code = NotFound desc = could not find container \"046b6dd409fb81acd39cd06f84189bbf263e7e0165243c376f25005134f786ec\": container with ID starting with 046b6dd409fb81acd39cd06f84189bbf263e7e0165243c376f25005134f786ec not found: ID does not exist" Oct 30 18:10:41.513564 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:41.513525 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-78d8ccc49c-8txzp"] Oct 30 18:10:41.527223 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:41.527185 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-78d8ccc49c-8txzp"] Oct 30 18:10:41.636306 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:41.636271 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0b1e9d0-b896-4a83-844d-d566d3d324d6" path="/var/lib/kubelet/pods/f0b1e9d0-b896-4a83-844d-d566d3d324d6/volumes" Oct 30 18:10:55.206396 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:10:55.206347 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c5884dffd-xh5zr"] Oct 30 18:11:00.208239 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:00.208173 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6c5884dffd-xh5zr" podUID="9554714f-5b63-4cec-80a1-e154f0d48142" containerName="payload-processing" containerID="cri-o://3dfe793828830fb90b8c533aa22471bf75a3a372a896194ae106504c0fa326cb" gracePeriod=25 Oct 30 18:11:00.395389 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:11:00.394940 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:11:00.494465 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:00.494435 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c5884dffd-xh5zr" Oct 30 18:11:00.587355 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:00.587302 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9554714f-5b63-4cec-80a1-e154f0d48142-kube-api-access-l5s9r\" (UniqueName: \"kubernetes.io/projected/9554714f-5b63-4cec-80a1-e154f0d48142-kube-api-access-l5s9r\") pod \"9554714f-5b63-4cec-80a1-e154f0d48142\" (UID: \"9554714f-5b63-4cec-80a1-e154f0d48142\") " Oct 30 18:11:00.587567 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:00.587517 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9554714f-5b63-4cec-80a1-e154f0d48142-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9554714f-5b63-4cec-80a1-e154f0d48142-plugins-config-volume\") pod \"9554714f-5b63-4cec-80a1-e154f0d48142\" (UID: \"9554714f-5b63-4cec-80a1-e154f0d48142\") " Oct 30 18:11:00.587567 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:00.587552 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9554714f-5b63-4cec-80a1-e154f0d48142-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9554714f-5b63-4cec-80a1-e154f0d48142-provider-credentials\") pod \"9554714f-5b63-4cec-80a1-e154f0d48142\" (UID: \"9554714f-5b63-4cec-80a1-e154f0d48142\") " Oct 30 18:11:00.587704 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:00.587686 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9554714f-5b63-4cec-80a1-e154f0d48142-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9554714f-5b63-4cec-80a1-e154f0d48142-routing-overlay-volume\") pod \"9554714f-5b63-4cec-80a1-e154f0d48142\" (UID: \"9554714f-5b63-4cec-80a1-e154f0d48142\") " Oct 30 18:11:00.589089 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:00.589060 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9554714f-5b63-4cec-80a1-e154f0d48142-routing-overlay-volume" pod "9554714f-5b63-4cec-80a1-e154f0d48142" (UID: "9554714f-5b63-4cec-80a1-e154f0d48142"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:11:00.589768 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:00.589744 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9554714f-5b63-4cec-80a1-e154f0d48142-plugins-config-volume" pod "9554714f-5b63-4cec-80a1-e154f0d48142" (UID: "9554714f-5b63-4cec-80a1-e154f0d48142"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:11:00.592293 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:00.592254 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9554714f-5b63-4cec-80a1-e154f0d48142-kube-api-access-l5s9r" pod "9554714f-5b63-4cec-80a1-e154f0d48142" (UID: "9554714f-5b63-4cec-80a1-e154f0d48142"). InnerVolumeSpecName "kube-api-access-l5s9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:11:00.604407 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:00.604359 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9554714f-5b63-4cec-80a1-e154f0d48142-provider-credentials" pod "9554714f-5b63-4cec-80a1-e154f0d48142" (UID: "9554714f-5b63-4cec-80a1-e154f0d48142"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:11:00.641935 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:00.641901 4455 generic.go:356] "Generic (PLEG): container finished" podID="9554714f-5b63-4cec-80a1-e154f0d48142" containerID="3dfe793828830fb90b8c533aa22471bf75a3a372a896194ae106504c0fa326cb" exitCode=0 Oct 30 18:11:00.642079 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:00.641953 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c5884dffd-xh5zr" Oct 30 18:11:00.642079 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:00.641989 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c5884dffd-xh5zr" event={"ID":"9554714f-5b63-4cec-80a1-e154f0d48142","Type":"ContainerDied","Data":"3dfe793828830fb90b8c533aa22471bf75a3a372a896194ae106504c0fa326cb"} Oct 30 18:11:00.642079 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:00.642046 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c5884dffd-xh5zr" event={"ID":"9554714f-5b63-4cec-80a1-e154f0d48142","Type":"ContainerDied","Data":"931bac847fe9c4188ff0950f56db249700398f250de9af5c4858c5a92176ade4"} Oct 30 18:11:00.642258 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:00.642074 4455 scope.go:122] "RemoveContainer" containerID="3dfe793828830fb90b8c533aa22471bf75a3a372a896194ae106504c0fa326cb" Oct 30 18:11:00.652844 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:00.652828 4455 scope.go:122] "RemoveContainer" containerID="3dfe793828830fb90b8c533aa22471bf75a3a372a896194ae106504c0fa326cb" Oct 30 18:11:00.653093 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:11:00.653073 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dfe793828830fb90b8c533aa22471bf75a3a372a896194ae106504c0fa326cb\": container with ID starting with 3dfe793828830fb90b8c533aa22471bf75a3a372a896194ae106504c0fa326cb not found: ID does not exist" containerID="3dfe793828830fb90b8c533aa22471bf75a3a372a896194ae106504c0fa326cb" Oct 30 18:11:00.653138 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:00.653101 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dfe793828830fb90b8c533aa22471bf75a3a372a896194ae106504c0fa326cb"} err="failed to get container status \"3dfe793828830fb90b8c533aa22471bf75a3a372a896194ae106504c0fa326cb\": rpc error: code = NotFound desc = could not find container \"3dfe793828830fb90b8c533aa22471bf75a3a372a896194ae106504c0fa326cb\": container with ID starting with 3dfe793828830fb90b8c533aa22471bf75a3a372a896194ae106504c0fa326cb not found: ID does not exist" Oct 30 18:11:00.690081 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:00.690042 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9554714f-5b63-4cec-80a1-e154f0d48142-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:11:00.690081 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:00.690078 4455 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9554714f-5b63-4cec-80a1-e154f0d48142-provider-credentials\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:11:00.690283 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:00.690093 4455 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9554714f-5b63-4cec-80a1-e154f0d48142-routing-overlay-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:11:00.690283 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:00.690110 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5s9r\" (UniqueName: \"kubernetes.io/projected/9554714f-5b63-4cec-80a1-e154f0d48142-kube-api-access-l5s9r\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:11:00.704838 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:00.704793 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c5884dffd-xh5zr"] Oct 30 18:11:00.710906 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:00.710838 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c5884dffd-xh5zr"] Oct 30 18:11:01.636256 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:01.636223 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9554714f-5b63-4cec-80a1-e154f0d48142" path="/var/lib/kubelet/pods/9554714f-5b63-4cec-80a1-e154f0d48142/volumes" Oct 30 18:11:26.342512 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:26.300989 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:11:27.354361 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:27.353976 4455 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:28.348461 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:28.348411 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:28.738278 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:28.725631 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:11:28.738278 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:28.725687 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" podUID="c7c21be6-b9eb-401c-aa3c-af5b62960cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:29.072349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:29.068620 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:11:29.072349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:29.068691 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" podUID="4e7b0c73-0041-4549-b47e-76a7aaed2b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:29.393413 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:29.380441 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:11:29.393413 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:29.381189 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:11:30.096354 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:30.095222 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-l4frw"] Oct 30 18:11:30.195869 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:30.195833 4455 cpu_manager.go:414] "containerMap: removing container" podUID="9554714f-5b63-4cec-80a1-e154f0d48142" containerName="payload-processing" Oct 30 18:11:30.196135 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:30.196114 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9554714f-5b63-4cec-80a1-e154f0d48142" containerName="payload-processing" Oct 30 18:11:30.196389 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:30.196370 4455 cpu_manager.go:414] "containerMap: removing container" podUID="f0b1e9d0-b896-4a83-844d-d566d3d324d6" containerName="payload-processing" Oct 30 18:11:30.196514 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:30.196500 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0b1e9d0-b896-4a83-844d-d566d3d324d6" containerName="payload-processing" Oct 30 18:11:30.196826 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:30.196809 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0b1e9d0-b896-4a83-844d-d566d3d324d6" containerName="payload-processing" Oct 30 18:11:30.196935 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:30.196923 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="9554714f-5b63-4cec-80a1-e154f0d48142" containerName="payload-processing" Oct 30 18:11:30.312653 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:30.312607 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-r6l4c"] Oct 30 18:11:30.339380 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:30.324472 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-l4frw" Oct 30 18:11:30.342177 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:30.342150 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c6b2120f-dockercfg-xtwdd\"" Oct 30 18:11:30.373885 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:30.373787 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c6b2120f\"" Oct 30 18:11:30.382440 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:30.382395 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:30.520354 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:30.509563 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-r6l4c"] Oct 30 18:11:30.530436 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:30.528569 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-r6l4c" Oct 30 18:11:30.606347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:30.587804 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-l4frw"] Oct 30 18:11:30.632405 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:30.617306 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txgqn\" (UniqueName: \"kubernetes.io/projected/0bd62dee-70b4-4f07-9f41-3186f5550228-kube-api-access-txgqn\") pod \"payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-l4frw\" (UID: \"0bd62dee-70b4-4f07-9f41-3186f5550228\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-l4frw" Oct 30 18:11:30.650538 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:30.650506 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd62dee-70b4-4f07-9f41-3186f5550228-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-l4frw\" (UID: \"0bd62dee-70b4-4f07-9f41-3186f5550228\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-l4frw" Oct 30 18:11:30.781358 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:30.772159 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr54b\" (UniqueName: \"kubernetes.io/projected/dcef406f-9a07-4803-b883-aac4b334de0c-kube-api-access-jr54b\") pod \"payload-processing-e2e-mt-c6b2120f-5666cb5864-r6l4c\" (UID: \"dcef406f-9a07-4803-b883-aac4b334de0c\") " pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-r6l4c" Oct 30 18:11:30.781702 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:30.781672 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd62dee-70b4-4f07-9f41-3186f5550228-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-l4frw\" (UID: \"0bd62dee-70b4-4f07-9f41-3186f5550228\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-l4frw" Oct 30 18:11:30.785352 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:30.781993 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcef406f-9a07-4803-b883-aac4b334de0c-plugins-config-volume\") pod \"payload-processing-e2e-mt-c6b2120f-5666cb5864-r6l4c\" (UID: \"dcef406f-9a07-4803-b883-aac4b334de0c\") " pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-r6l4c" Oct 30 18:11:30.789347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:30.785664 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txgqn\" (UniqueName: \"kubernetes.io/projected/0bd62dee-70b4-4f07-9f41-3186f5550228-kube-api-access-txgqn\") pod \"payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-l4frw\" (UID: \"0bd62dee-70b4-4f07-9f41-3186f5550228\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-l4frw" Oct 30 18:11:30.839361 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:30.815884 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd62dee-70b4-4f07-9f41-3186f5550228-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-l4frw\" (UID: \"0bd62dee-70b4-4f07-9f41-3186f5550228\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-l4frw" Oct 30 18:11:30.958406 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:30.948486 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcef406f-9a07-4803-b883-aac4b334de0c-plugins-config-volume\") pod \"payload-processing-e2e-mt-c6b2120f-5666cb5864-r6l4c\" (UID: \"dcef406f-9a07-4803-b883-aac4b334de0c\") " pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-r6l4c" Oct 30 18:11:30.958406 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:30.948741 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr54b\" (UniqueName: \"kubernetes.io/projected/dcef406f-9a07-4803-b883-aac4b334de0c-kube-api-access-jr54b\") pod \"payload-processing-e2e-mt-c6b2120f-5666cb5864-r6l4c\" (UID: \"dcef406f-9a07-4803-b883-aac4b334de0c\") " pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-r6l4c" Oct 30 18:11:30.958406 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:30.950197 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcef406f-9a07-4803-b883-aac4b334de0c-plugins-config-volume\") pod \"payload-processing-e2e-mt-c6b2120f-5666cb5864-r6l4c\" (UID: \"dcef406f-9a07-4803-b883-aac4b334de0c\") " pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-r6l4c" Oct 30 18:11:31.008419 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:31.001281 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txgqn\" (UniqueName: \"kubernetes.io/projected/0bd62dee-70b4-4f07-9f41-3186f5550228-kube-api-access-txgqn\") pod \"payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-l4frw\" (UID: \"0bd62dee-70b4-4f07-9f41-3186f5550228\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-l4frw" Oct 30 18:11:31.064640 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:31.064601 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-l4frw" Oct 30 18:11:31.186562 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:11:31.186520 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:11:31.242175 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:31.225676 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:11:31.242175 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:31.225746 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" podUID="1d3471dc-aaf3-4060-a64d-2413e522e65a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:31.401360 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:31.390592 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr54b\" (UniqueName: \"kubernetes.io/projected/dcef406f-9a07-4803-b883-aac4b334de0c-kube-api-access-jr54b\") pod \"payload-processing-e2e-mt-c6b2120f-5666cb5864-r6l4c\" (UID: \"dcef406f-9a07-4803-b883-aac4b334de0c\") " pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-r6l4c" Oct 30 18:11:31.401667 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:31.391519 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:31.445248 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:31.445209 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-r6l4c" Oct 30 18:11:32.441894 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:32.441744 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:32.697520 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:32.696798 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:11:33.042308 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:33.042177 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-r6l4c"] Oct 30 18:11:33.116752 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:33.116246 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-l4frw"] Oct 30 18:11:33.856868 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:33.856263 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-r6l4c" event={"ID":"dcef406f-9a07-4803-b883-aac4b334de0c","Type":"ContainerStarted","Data":"470c7e5dec6fb249ce2a899131286afb4aa46ceeba34fca23bfa4c6cbfce8eb5"} Oct 30 18:11:33.858494 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:33.857403 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-r6l4c" event={"ID":"dcef406f-9a07-4803-b883-aac4b334de0c","Type":"ContainerStarted","Data":"a33eb1d5c4d2889ce490c5211203c1bb4f53fa34c3e3025cfdf5852863735d65"} Oct 30 18:11:33.858726 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:33.858702 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-r6l4c" Oct 30 18:11:33.861507 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:33.861426 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-l4frw" event={"ID":"0bd62dee-70b4-4f07-9f41-3186f5550228","Type":"ContainerStarted","Data":"cdf57d5e26ad49d89da6fbc94fadefb5199024081ac5a9943365e222c42636f2"} Oct 30 18:11:33.861507 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:33.861471 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-l4frw" event={"ID":"0bd62dee-70b4-4f07-9f41-3186f5550228","Type":"ContainerStarted","Data":"2e5121796164a7a15bfd78d3fc33c0578559b9cf9212f35f22852de46857d1fa"} Oct 30 18:11:33.909410 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:33.907333 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-r6l4c" podStartSLOduration=4.907298772 podStartE2EDuration="4.907298772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:33.904091144 +0000 UTC m=+1146.391728180" watchObservedRunningTime="2026-10-30 18:11:33.907298772 +0000 UTC m=+1146.394935808" Oct 30 18:11:33.957882 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:33.956633 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-l4frw" podStartSLOduration=4.956613581 podStartE2EDuration="4.956613581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:33.954858659 +0000 UTC m=+1146.442495694" watchObservedRunningTime="2026-10-30 18:11:33.956613581 +0000 UTC m=+1146.444250619" Oct 30 18:11:34.873463 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:34.873416 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-l4frw" Oct 30 18:11:41.887747 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:41.887712 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq"] Oct 30 18:11:41.891682 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:41.891662 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" Oct 30 18:11:41.919585 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:41.919535 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c6b2120f-serving-cert\"" Oct 30 18:11:41.927690 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:41.927656 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq"] Oct 30 18:11:41.952094 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:41.952043 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bf6d8f4-3ec1-449f-9399-e2e238d5ee11-maas-api-tls\") pod \"maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq\" (UID: \"9bf6d8f4-3ec1-449f-9399-e2e238d5ee11\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" Oct 30 18:11:41.952501 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:41.952379 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwv24\" (UniqueName: \"kubernetes.io/projected/9bf6d8f4-3ec1-449f-9399-e2e238d5ee11-kube-api-access-nwv24\") pod \"maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq\" (UID: \"9bf6d8f4-3ec1-449f-9399-e2e238d5ee11\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" Oct 30 18:11:41.952782 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:41.952761 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bf6d8f4-3ec1-449f-9399-e2e238d5ee11-openshift-service-ca\") pod \"maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq\" (UID: \"9bf6d8f4-3ec1-449f-9399-e2e238d5ee11\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" Oct 30 18:11:41.953454 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:41.953430 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bf6d8f4-3ec1-449f-9399-e2e238d5ee11-metrics-certs\") pod \"maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq\" (UID: \"9bf6d8f4-3ec1-449f-9399-e2e238d5ee11\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" Oct 30 18:11:42.055621 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:42.055587 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bf6d8f4-3ec1-449f-9399-e2e238d5ee11-metrics-certs\") pod \"maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq\" (UID: \"9bf6d8f4-3ec1-449f-9399-e2e238d5ee11\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" Oct 30 18:11:42.055834 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:42.055811 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bf6d8f4-3ec1-449f-9399-e2e238d5ee11-maas-api-tls\") pod \"maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq\" (UID: \"9bf6d8f4-3ec1-449f-9399-e2e238d5ee11\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" Oct 30 18:11:42.056163 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:42.056138 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwv24\" (UniqueName: \"kubernetes.io/projected/9bf6d8f4-3ec1-449f-9399-e2e238d5ee11-kube-api-access-nwv24\") pod \"maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq\" (UID: \"9bf6d8f4-3ec1-449f-9399-e2e238d5ee11\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" Oct 30 18:11:42.056487 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:42.056467 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bf6d8f4-3ec1-449f-9399-e2e238d5ee11-openshift-service-ca\") pod \"maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq\" (UID: \"9bf6d8f4-3ec1-449f-9399-e2e238d5ee11\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" Oct 30 18:11:42.071647 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:11:42.056094 4455 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-serving-cert: secret "maas-api-e2e-mt-c6b2120f-serving-cert" not found Oct 30 18:11:42.072250 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:11:42.072226 4455 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bf6d8f4-3ec1-449f-9399-e2e238d5ee11-maas-api-tls podName:9bf6d8f4-3ec1-449f-9399-e2e238d5ee11 nodeName:}" failed. No retries permitted until 2026-10-30 18:11:42.572164099 +0000 UTC m=+1155.059801131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9bf6d8f4-3ec1-449f-9399-e2e238d5ee11-maas-api-tls") pod "maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" (UID: "9bf6d8f4-3ec1-449f-9399-e2e238d5ee11") : secret "maas-api-e2e-mt-c6b2120f-serving-cert" not found Oct 30 18:11:42.072535 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:42.059486 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bf6d8f4-3ec1-449f-9399-e2e238d5ee11-metrics-certs\") pod \"maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq\" (UID: \"9bf6d8f4-3ec1-449f-9399-e2e238d5ee11\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" Oct 30 18:11:42.072716 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:42.057639 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bf6d8f4-3ec1-449f-9399-e2e238d5ee11-openshift-service-ca\") pod \"maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq\" (UID: \"9bf6d8f4-3ec1-449f-9399-e2e238d5ee11\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" Oct 30 18:11:42.099227 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:42.099152 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwv24\" (UniqueName: \"kubernetes.io/projected/9bf6d8f4-3ec1-449f-9399-e2e238d5ee11-kube-api-access-nwv24\") pod \"maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq\" (UID: \"9bf6d8f4-3ec1-449f-9399-e2e238d5ee11\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" Oct 30 18:11:42.580152 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:42.580116 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bf6d8f4-3ec1-449f-9399-e2e238d5ee11-maas-api-tls\") pod \"maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq\" (UID: \"9bf6d8f4-3ec1-449f-9399-e2e238d5ee11\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" Oct 30 18:11:42.582995 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:42.582965 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bf6d8f4-3ec1-449f-9399-e2e238d5ee11-maas-api-tls\") pod \"maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq\" (UID: \"9bf6d8f4-3ec1-449f-9399-e2e238d5ee11\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" Oct 30 18:11:42.810343 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:42.810300 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" Oct 30 18:11:43.184632 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:43.184598 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq"] Oct 30 18:11:43.185119 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:43.184830 4455 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:11:43.992294 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:43.992259 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" event={"ID":"9bf6d8f4-3ec1-449f-9399-e2e238d5ee11","Type":"ContainerStarted","Data":"c5e0e104cdd420e8848974cf913657dff7f05adc2cd110973c808eee2b6975a1"} Oct 30 18:11:43.992516 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:43.992317 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" event={"ID":"9bf6d8f4-3ec1-449f-9399-e2e238d5ee11","Type":"ContainerStarted","Data":"62291a4d69baa46e410831d1e4e6fdd4b08340bdf99959b270137579bbc43e1b"} Oct 30 18:11:43.992516 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:43.992497 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" Oct 30 18:11:44.039359 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:44.039240 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" podStartSLOduration=2.56733654 podStartE2EDuration="3.039226392s" totalImagesPullingTime="471.889852ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:44.037634914 +0000 UTC m=+1156.525271951" watchObservedRunningTime="2026-10-30 18:11:44.039226392 +0000 UTC m=+1156.526863425" Oct 30 18:11:45.887282 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:45.887215 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-l4frw" Oct 30 18:11:45.887282 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:45.887288 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-r6l4c" Oct 30 18:11:50.001826 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:50.001777 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" podUID="9bf6d8f4-3ec1-449f-9399-e2e238d5ee11" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.106:8443/health\": dial tcp 10.134.0.106:8443: connect: connection refused" Oct 30 18:11:54.120207 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:54.120110 4455 generic.go:356] "Generic (PLEG): container finished" podID="9bf6d8f4-3ec1-449f-9399-e2e238d5ee11" containerID="c5e0e104cdd420e8848974cf913657dff7f05adc2cd110973c808eee2b6975a1" exitCode=1 Oct 30 18:11:54.120709 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:54.120353 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" event={"ID":"9bf6d8f4-3ec1-449f-9399-e2e238d5ee11","Type":"ContainerDied","Data":"c5e0e104cdd420e8848974cf913657dff7f05adc2cd110973c808eee2b6975a1"} Oct 30 18:11:54.122048 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:54.120942 4455 scope.go:122] "RemoveContainer" containerID="c5e0e104cdd420e8848974cf913657dff7f05adc2cd110973c808eee2b6975a1" Oct 30 18:11:55.130464 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:55.130428 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" event={"ID":"9bf6d8f4-3ec1-449f-9399-e2e238d5ee11","Type":"ContainerStarted","Data":"48a9d314fbb81eaaef69139936147e7b16c22d2d0fda244bfd11d40f0aecb326"} Oct 30 18:11:55.130918 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:55.130649 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" Oct 30 18:11:59.407742 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:59.407701 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c5891f-5586d8fd8c-l2xvx"] Oct 30 18:11:59.412446 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:59.412419 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5891f-5586d8fd8c-l2xvx" Oct 30 18:11:59.431248 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:59.431225 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c5891f\"" Oct 30 18:11:59.436362 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:59.436309 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c5891f-5586d8fd8c-l2xvx"] Oct 30 18:11:59.444968 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:59.444947 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c5891f-dockercfg-4f9xn\"" Oct 30 18:11:59.502052 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:59.502023 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c5891f-7d94c4cd4b-b2w76"] Oct 30 18:11:59.507000 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:59.506974 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c5891f-7d94c4cd4b-b2w76" Oct 30 18:11:59.709388 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:59.707477 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b669d0f-5909-4120-87a8-f90ef36cd84b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c5891f-7d94c4cd4b-b2w76\" (UID: \"4b669d0f-5909-4120-87a8-f90ef36cd84b\") " pod="openshift-ingress/payload-processing-e2e-api-a-c5891f-7d94c4cd4b-b2w76" Oct 30 18:11:59.709388 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:59.707595 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc8a16fa-fd53-4372-967d-f2f556035841-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c5891f-5586d8fd8c-l2xvx\" (UID: \"cc8a16fa-fd53-4372-967d-f2f556035841\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5891f-5586d8fd8c-l2xvx" Oct 30 18:11:59.709388 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:59.707660 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmx8p\" (UniqueName: \"kubernetes.io/projected/4b669d0f-5909-4120-87a8-f90ef36cd84b-kube-api-access-bmx8p\") pod \"payload-processing-e2e-api-a-c5891f-7d94c4cd4b-b2w76\" (UID: \"4b669d0f-5909-4120-87a8-f90ef36cd84b\") " pod="openshift-ingress/payload-processing-e2e-api-a-c5891f-7d94c4cd4b-b2w76" Oct 30 18:11:59.709388 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:59.707881 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7kkw\" (UniqueName: \"kubernetes.io/projected/cc8a16fa-fd53-4372-967d-f2f556035841-kube-api-access-j7kkw\") pod \"payload-pre-processing-e2e-api-a-c5891f-5586d8fd8c-l2xvx\" (UID: \"cc8a16fa-fd53-4372-967d-f2f556035841\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5891f-5586d8fd8c-l2xvx" Oct 30 18:11:59.729383 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:59.720901 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c5891f-7d94c4cd4b-b2w76"] Oct 30 18:11:59.813532 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:59.813490 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7kkw\" (UniqueName: \"kubernetes.io/projected/cc8a16fa-fd53-4372-967d-f2f556035841-kube-api-access-j7kkw\") pod \"payload-pre-processing-e2e-api-a-c5891f-5586d8fd8c-l2xvx\" (UID: \"cc8a16fa-fd53-4372-967d-f2f556035841\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5891f-5586d8fd8c-l2xvx" Oct 30 18:11:59.813753 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:59.813573 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b669d0f-5909-4120-87a8-f90ef36cd84b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c5891f-7d94c4cd4b-b2w76\" (UID: \"4b669d0f-5909-4120-87a8-f90ef36cd84b\") " pod="openshift-ingress/payload-processing-e2e-api-a-c5891f-7d94c4cd4b-b2w76" Oct 30 18:11:59.813753 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:59.813706 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc8a16fa-fd53-4372-967d-f2f556035841-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c5891f-5586d8fd8c-l2xvx\" (UID: \"cc8a16fa-fd53-4372-967d-f2f556035841\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5891f-5586d8fd8c-l2xvx" Oct 30 18:11:59.813889 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:59.813779 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmx8p\" (UniqueName: \"kubernetes.io/projected/4b669d0f-5909-4120-87a8-f90ef36cd84b-kube-api-access-bmx8p\") pod \"payload-processing-e2e-api-a-c5891f-7d94c4cd4b-b2w76\" (UID: \"4b669d0f-5909-4120-87a8-f90ef36cd84b\") " pod="openshift-ingress/payload-processing-e2e-api-a-c5891f-7d94c4cd4b-b2w76" Oct 30 18:11:59.815082 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:59.815059 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc8a16fa-fd53-4372-967d-f2f556035841-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c5891f-5586d8fd8c-l2xvx\" (UID: \"cc8a16fa-fd53-4372-967d-f2f556035841\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5891f-5586d8fd8c-l2xvx" Oct 30 18:11:59.816124 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:59.816102 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b669d0f-5909-4120-87a8-f90ef36cd84b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c5891f-7d94c4cd4b-b2w76\" (UID: \"4b669d0f-5909-4120-87a8-f90ef36cd84b\") " pod="openshift-ingress/payload-processing-e2e-api-a-c5891f-7d94c4cd4b-b2w76" Oct 30 18:11:59.839725 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:59.839694 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7kkw\" (UniqueName: \"kubernetes.io/projected/cc8a16fa-fd53-4372-967d-f2f556035841-kube-api-access-j7kkw\") pod \"payload-pre-processing-e2e-api-a-c5891f-5586d8fd8c-l2xvx\" (UID: \"cc8a16fa-fd53-4372-967d-f2f556035841\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5891f-5586d8fd8c-l2xvx" Oct 30 18:11:59.839890 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:11:59.839782 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmx8p\" (UniqueName: \"kubernetes.io/projected/4b669d0f-5909-4120-87a8-f90ef36cd84b-kube-api-access-bmx8p\") pod \"payload-processing-e2e-api-a-c5891f-7d94c4cd4b-b2w76\" (UID: \"4b669d0f-5909-4120-87a8-f90ef36cd84b\") " pod="openshift-ingress/payload-processing-e2e-api-a-c5891f-7d94c4cd4b-b2w76" Oct 30 18:12:00.036201 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:00.036165 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5891f-5586d8fd8c-l2xvx" Oct 30 18:12:00.050680 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:00.050657 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c5891f-7d94c4cd4b-b2w76" Oct 30 18:12:00.352112 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:12:00.352083 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc8a16fa_fd53_4372_967d_f2f556035841.slice/crio-2b3280029dc5c9f7fd8c92f78163cef6838375da9fc88d54286f517b005921e7 WatchSource:0}: Error finding container 2b3280029dc5c9f7fd8c92f78163cef6838375da9fc88d54286f517b005921e7: Status 404 returned error can't find the container with id 2b3280029dc5c9f7fd8c92f78163cef6838375da9fc88d54286f517b005921e7 Oct 30 18:12:00.353482 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:00.353457 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c5891f-5586d8fd8c-l2xvx"] Oct 30 18:12:00.381738 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:00.381705 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c5891f-7d94c4cd4b-b2w76"] Oct 30 18:12:00.398031 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:12:00.396295 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:12:01.149453 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:01.149416 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" podUID="9bf6d8f4-3ec1-449f-9399-e2e238d5ee11" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.106:8443/health\": dial tcp 10.134.0.106:8443: connect: connection refused" Oct 30 18:12:01.193103 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:01.193066 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c5891f-7d94c4cd4b-b2w76" event={"ID":"4b669d0f-5909-4120-87a8-f90ef36cd84b","Type":"ContainerStarted","Data":"9e2e2cefa3175c33b620404054ac1e426f6f30153fb6cb23d3e4d6b7c8456130"} Oct 30 18:12:01.193305 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:01.193116 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c5891f-7d94c4cd4b-b2w76" event={"ID":"4b669d0f-5909-4120-87a8-f90ef36cd84b","Type":"ContainerStarted","Data":"526fe277e77ad7f27b47dfd0f00e39e9038d8baefb8cb87318c03df2100ad0eb"} Oct 30 18:12:01.193305 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:01.193219 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c5891f-7d94c4cd4b-b2w76" Oct 30 18:12:01.194781 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:01.194758 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5891f-5586d8fd8c-l2xvx" event={"ID":"cc8a16fa-fd53-4372-967d-f2f556035841","Type":"ContainerStarted","Data":"471fcb0f6f4881912895dab711929c9488b4770c1676b700f6c6e64c7307649c"} Oct 30 18:12:01.194885 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:01.194797 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5891f-5586d8fd8c-l2xvx" event={"ID":"cc8a16fa-fd53-4372-967d-f2f556035841","Type":"ContainerStarted","Data":"2b3280029dc5c9f7fd8c92f78163cef6838375da9fc88d54286f517b005921e7"} Oct 30 18:12:01.194927 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:01.194900 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5891f-5586d8fd8c-l2xvx" Oct 30 18:12:01.240495 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:01.240441 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c5891f-7d94c4cd4b-b2w76" podStartSLOduration=2.240427296 podStartE2EDuration="2.240427296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:01.237019213 +0000 UTC m=+1173.724656246" watchObservedRunningTime="2026-10-30 18:12:01.240427296 +0000 UTC m=+1173.728064329" Oct 30 18:12:01.285399 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:01.285340 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5891f-5586d8fd8c-l2xvx" podStartSLOduration=2.28530304 podStartE2EDuration="2.28530304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:01.280606434 +0000 UTC m=+1173.768243470" watchObservedRunningTime="2026-10-30 18:12:01.28530304 +0000 UTC m=+1173.772940075" Oct 30 18:12:05.229819 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:05.229785 4455 generic.go:356] "Generic (PLEG): container finished" podID="9bf6d8f4-3ec1-449f-9399-e2e238d5ee11" containerID="48a9d314fbb81eaaef69139936147e7b16c22d2d0fda244bfd11d40f0aecb326" exitCode=1 Oct 30 18:12:05.230369 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:05.229858 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" event={"ID":"9bf6d8f4-3ec1-449f-9399-e2e238d5ee11","Type":"ContainerDied","Data":"48a9d314fbb81eaaef69139936147e7b16c22d2d0fda244bfd11d40f0aecb326"} Oct 30 18:12:05.230369 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:05.229934 4455 scope.go:122] "RemoveContainer" containerID="c5e0e104cdd420e8848974cf913657dff7f05adc2cd110973c808eee2b6975a1" Oct 30 18:12:05.230519 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:05.230391 4455 scope.go:122] "RemoveContainer" containerID="48a9d314fbb81eaaef69139936147e7b16c22d2d0fda244bfd11d40f0aecb326" Oct 30 18:12:05.230749 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:12:05.230729 4455 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq_odh-ai-gateway-infra(9bf6d8f4-3ec1-449f-9399-e2e238d5ee11)\"" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" podUID="9bf6d8f4-3ec1-449f-9399-e2e238d5ee11" Oct 30 18:12:09.086193 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:09.086157 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks"] Oct 30 18:12:09.093489 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:09.093466 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks" Oct 30 18:12:09.118160 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:09.118132 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks"] Oct 30 18:12:09.127369 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:09.122590 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c5891f-serving-cert\"" Oct 30 18:12:09.170888 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:09.170856 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e346787b-4f18-4533-ba53-86c319761600-metrics-certs\") pod \"maas-api-e2e-api-a-c5891f-8798fdcd-5kwks\" (UID: \"e346787b-4f18-4533-ba53-86c319761600\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks" Oct 30 18:12:09.171058 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:09.170974 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e346787b-4f18-4533-ba53-86c319761600-maas-api-tls\") pod \"maas-api-e2e-api-a-c5891f-8798fdcd-5kwks\" (UID: \"e346787b-4f18-4533-ba53-86c319761600\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks" Oct 30 18:12:09.171126 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:09.171106 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e346787b-4f18-4533-ba53-86c319761600-openshift-service-ca\") pod \"maas-api-e2e-api-a-c5891f-8798fdcd-5kwks\" (UID: \"e346787b-4f18-4533-ba53-86c319761600\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks" Oct 30 18:12:09.171339 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:09.171295 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fx2q\" (UniqueName: \"kubernetes.io/projected/e346787b-4f18-4533-ba53-86c319761600-kube-api-access-4fx2q\") pod \"maas-api-e2e-api-a-c5891f-8798fdcd-5kwks\" (UID: \"e346787b-4f18-4533-ba53-86c319761600\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks" Oct 30 18:12:09.274081 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:09.273054 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e346787b-4f18-4533-ba53-86c319761600-metrics-certs\") pod \"maas-api-e2e-api-a-c5891f-8798fdcd-5kwks\" (UID: \"e346787b-4f18-4533-ba53-86c319761600\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks" Oct 30 18:12:09.274081 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:09.273173 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e346787b-4f18-4533-ba53-86c319761600-maas-api-tls\") pod \"maas-api-e2e-api-a-c5891f-8798fdcd-5kwks\" (UID: \"e346787b-4f18-4533-ba53-86c319761600\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks" Oct 30 18:12:09.274081 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:09.273287 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e346787b-4f18-4533-ba53-86c319761600-openshift-service-ca\") pod \"maas-api-e2e-api-a-c5891f-8798fdcd-5kwks\" (UID: \"e346787b-4f18-4533-ba53-86c319761600\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks" Oct 30 18:12:09.274081 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:09.273409 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fx2q\" (UniqueName: \"kubernetes.io/projected/e346787b-4f18-4533-ba53-86c319761600-kube-api-access-4fx2q\") pod \"maas-api-e2e-api-a-c5891f-8798fdcd-5kwks\" (UID: \"e346787b-4f18-4533-ba53-86c319761600\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks" Oct 30 18:12:09.276071 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:09.274523 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e346787b-4f18-4533-ba53-86c319761600-openshift-service-ca\") pod \"maas-api-e2e-api-a-c5891f-8798fdcd-5kwks\" (UID: \"e346787b-4f18-4533-ba53-86c319761600\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks" Oct 30 18:12:09.283366 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:09.282060 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e346787b-4f18-4533-ba53-86c319761600-metrics-certs\") pod \"maas-api-e2e-api-a-c5891f-8798fdcd-5kwks\" (UID: \"e346787b-4f18-4533-ba53-86c319761600\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks" Oct 30 18:12:09.286368 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:09.284790 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e346787b-4f18-4533-ba53-86c319761600-maas-api-tls\") pod \"maas-api-e2e-api-a-c5891f-8798fdcd-5kwks\" (UID: \"e346787b-4f18-4533-ba53-86c319761600\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks" Oct 30 18:12:09.298753 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:09.298726 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fx2q\" (UniqueName: \"kubernetes.io/projected/e346787b-4f18-4533-ba53-86c319761600-kube-api-access-4fx2q\") pod \"maas-api-e2e-api-a-c5891f-8798fdcd-5kwks\" (UID: \"e346787b-4f18-4533-ba53-86c319761600\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks" Oct 30 18:12:09.409124 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:09.409032 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks" Oct 30 18:12:09.771290 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:12:09.771247 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode346787b_4f18_4533_ba53_86c319761600.slice/crio-0d7e445225eef4cbe6128ef3cfecb3f4d3d6005d4a13007bbd8effaa0b27afaa WatchSource:0}: Error finding container 0d7e445225eef4cbe6128ef3cfecb3f4d3d6005d4a13007bbd8effaa0b27afaa: Status 404 returned error can't find the container with id 0d7e445225eef4cbe6128ef3cfecb3f4d3d6005d4a13007bbd8effaa0b27afaa Oct 30 18:12:09.772934 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:09.772910 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks"] Oct 30 18:12:10.395483 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:10.395257 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks" event={"ID":"e346787b-4f18-4533-ba53-86c319761600","Type":"ContainerStarted","Data":"cf185c5054f7d01a7b74716b0dac4ea1e808e0df66ad5c435b3eb282a57ed79d"} Oct 30 18:12:10.395483 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:10.395316 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks" event={"ID":"e346787b-4f18-4533-ba53-86c319761600","Type":"ContainerStarted","Data":"0d7e445225eef4cbe6128ef3cfecb3f4d3d6005d4a13007bbd8effaa0b27afaa"} Oct 30 18:12:10.395483 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:10.395378 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks" Oct 30 18:12:10.444866 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:10.444811 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks" podStartSLOduration=1.127055263 podStartE2EDuration="1.44479849s" totalImagesPullingTime="317.743227ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:10.441457279 +0000 UTC m=+1182.929094336" watchObservedRunningTime="2026-10-30 18:12:10.44479849 +0000 UTC m=+1182.932435524" Oct 30 18:12:12.206284 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:12.206255 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5891f-5586d8fd8c-l2xvx" Oct 30 18:12:12.206706 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:12.206359 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c5891f-7d94c4cd4b-b2w76" Oct 30 18:12:12.810744 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:12.810710 4455 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" Oct 30 18:12:12.811301 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:12.811280 4455 scope.go:122] "RemoveContainer" containerID="48a9d314fbb81eaaef69139936147e7b16c22d2d0fda244bfd11d40f0aecb326" Oct 30 18:12:12.811776 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:12:12.811747 4455 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq_odh-ai-gateway-infra(9bf6d8f4-3ec1-449f-9399-e2e238d5ee11)\"" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" podUID="9bf6d8f4-3ec1-449f-9399-e2e238d5ee11" Oct 30 18:12:15.627621 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:15.627590 4455 scope.go:122] "RemoveContainer" containerID="48a9d314fbb81eaaef69139936147e7b16c22d2d0fda244bfd11d40f0aecb326" Oct 30 18:12:16.404825 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:16.404725 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks" podUID="e346787b-4f18-4533-ba53-86c319761600" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.109:8443/health\": dial tcp 10.134.0.109:8443: connect: connection refused" Oct 30 18:12:16.454915 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:16.454873 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" event={"ID":"9bf6d8f4-3ec1-449f-9399-e2e238d5ee11","Type":"ContainerStarted","Data":"199472a03476a3106ef72b622a543763804d52f8cdd439a718fb52cb6810c28a"} Oct 30 18:12:16.455152 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:16.455128 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" Oct 30 18:12:20.489676 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:20.489637 4455 generic.go:356] "Generic (PLEG): container finished" podID="e346787b-4f18-4533-ba53-86c319761600" containerID="cf185c5054f7d01a7b74716b0dac4ea1e808e0df66ad5c435b3eb282a57ed79d" exitCode=1 Oct 30 18:12:20.490081 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:20.489717 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks" event={"ID":"e346787b-4f18-4533-ba53-86c319761600","Type":"ContainerDied","Data":"cf185c5054f7d01a7b74716b0dac4ea1e808e0df66ad5c435b3eb282a57ed79d"} Oct 30 18:12:20.490253 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:20.490236 4455 scope.go:122] "RemoveContainer" containerID="cf185c5054f7d01a7b74716b0dac4ea1e808e0df66ad5c435b3eb282a57ed79d" Oct 30 18:12:21.500876 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:21.500839 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks" event={"ID":"e346787b-4f18-4533-ba53-86c319761600","Type":"ContainerStarted","Data":"5fb6bdc9436f2012a26ff7c9edf610ec5c9eed20689e530f04469e109516770d"} Oct 30 18:12:21.501408 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:21.501063 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks" Oct 30 18:12:22.464584 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:22.464547 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" podUID="9bf6d8f4-3ec1-449f-9399-e2e238d5ee11" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.106:8443/health\": dial tcp 10.134.0.106:8443: connect: connection refused" Oct 30 18:12:26.436445 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:26.436398 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:27.347404 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:27.346518 4455 generic.go:356] "Generic (PLEG): container finished" podID="9bf6d8f4-3ec1-449f-9399-e2e238d5ee11" containerID="199472a03476a3106ef72b622a543763804d52f8cdd439a718fb52cb6810c28a" exitCode=1 Oct 30 18:12:27.347404 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:27.346783 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" event={"ID":"9bf6d8f4-3ec1-449f-9399-e2e238d5ee11","Type":"ContainerDied","Data":"199472a03476a3106ef72b622a543763804d52f8cdd439a718fb52cb6810c28a"} Oct 30 18:12:27.347404 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:27.346861 4455 scope.go:122] "RemoveContainer" containerID="48a9d314fbb81eaaef69139936147e7b16c22d2d0fda244bfd11d40f0aecb326" Oct 30 18:12:27.347740 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:27.347613 4455 scope.go:122] "RemoveContainer" containerID="199472a03476a3106ef72b622a543763804d52f8cdd439a718fb52cb6810c28a" Oct 30 18:12:27.348163 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:12:27.348133 4455 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq_odh-ai-gateway-infra(9bf6d8f4-3ec1-449f-9399-e2e238d5ee11)\"" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" podUID="9bf6d8f4-3ec1-449f-9399-e2e238d5ee11" Oct 30 18:12:27.606400 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:27.605479 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks" podUID="e346787b-4f18-4533-ba53-86c319761600" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.109:8443/health\": dial tcp 10.134.0.109:8443: connect: connection refused" Oct 30 18:12:29.010351 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:29.004882 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:12:29.010351 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:29.004950 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" podUID="4e7b0c73-0041-4549-b47e-76a7aaed2b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:31.235538 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:31.235478 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:12:31.265495 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:31.239421 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" podUID="1d3471dc-aaf3-4060-a64d-2413e522e65a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:31.313452 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:12:31.293924 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:12:31.420103 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:31.420059 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:32.352067 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:32.352025 4455 generic.go:356] "Generic (PLEG): container finished" podID="e346787b-4f18-4533-ba53-86c319761600" containerID="5fb6bdc9436f2012a26ff7c9edf610ec5c9eed20689e530f04469e109516770d" exitCode=1 Oct 30 18:12:32.373348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:32.368917 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks" event={"ID":"e346787b-4f18-4533-ba53-86c319761600","Type":"ContainerDied","Data":"5fb6bdc9436f2012a26ff7c9edf610ec5c9eed20689e530f04469e109516770d"} Oct 30 18:12:32.373348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:32.369015 4455 scope.go:122] "RemoveContainer" containerID="cf185c5054f7d01a7b74716b0dac4ea1e808e0df66ad5c435b3eb282a57ed79d" Oct 30 18:12:32.373348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:32.369666 4455 scope.go:122] "RemoveContainer" containerID="5fb6bdc9436f2012a26ff7c9edf610ec5c9eed20689e530f04469e109516770d" Oct 30 18:12:32.373348 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:12:32.370082 4455 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-api-a-c5891f-8798fdcd-5kwks_odh-ai-gateway-infra(e346787b-4f18-4533-ba53-86c319761600)\"" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks" podUID="e346787b-4f18-4533-ba53-86c319761600" Oct 30 18:12:32.812348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:32.811419 4455 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" Oct 30 18:12:32.812348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:32.811961 4455 scope.go:122] "RemoveContainer" containerID="199472a03476a3106ef72b622a543763804d52f8cdd439a718fb52cb6810c28a" Oct 30 18:12:32.812627 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:12:32.812389 4455 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq_odh-ai-gateway-infra(9bf6d8f4-3ec1-449f-9399-e2e238d5ee11)\"" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" podUID="9bf6d8f4-3ec1-449f-9399-e2e238d5ee11" Oct 30 18:12:33.115538 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:33.115435 4455 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:12:33.115799 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:33.115768 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" podUID="d4e8e077-c9f7-44c8-9dd5-d1c7a2403862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:33.429008 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:33.428896 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:12:39.413130 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:39.413040 4455 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks" Oct 30 18:12:39.413756 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:39.413576 4455 scope.go:122] "RemoveContainer" containerID="5fb6bdc9436f2012a26ff7c9edf610ec5c9eed20689e530f04469e109516770d" Oct 30 18:12:39.413952 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:12:39.413930 4455 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-api-a-c5891f-8798fdcd-5kwks_odh-ai-gateway-infra(e346787b-4f18-4533-ba53-86c319761600)\"" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks" podUID="e346787b-4f18-4533-ba53-86c319761600" Oct 30 18:12:41.627706 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:41.627678 4455 scope.go:122] "RemoveContainer" containerID="5fb6bdc9436f2012a26ff7c9edf610ec5c9eed20689e530f04469e109516770d" Oct 30 18:12:42.595544 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:42.595508 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks" event={"ID":"e346787b-4f18-4533-ba53-86c319761600","Type":"ContainerStarted","Data":"888a3db4a4b86f701d0675e5eae1984c2ada3f601450f8afb3d2d42507878b6e"} Oct 30 18:12:42.595752 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:42.595736 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks" Oct 30 18:12:48.610060 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:48.610025 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks" Oct 30 18:12:49.028689 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:49.028654 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb"] Oct 30 18:12:49.033109 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:49.033082 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb" Oct 30 18:12:49.051904 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:49.051868 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-4c8e1a-serving-cert\"" Oct 30 18:12:49.071230 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:49.071192 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec7ae614-bd4b-45a4-862e-1f499ddadbd6-metrics-certs\") pod \"maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb\" (UID: \"ec7ae614-bd4b-45a4-862e-1f499ddadbd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb" Oct 30 18:12:49.071764 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:49.071736 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec7ae614-bd4b-45a4-862e-1f499ddadbd6-maas-api-tls\") pod \"maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb\" (UID: \"ec7ae614-bd4b-45a4-862e-1f499ddadbd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb" Oct 30 18:12:49.071992 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:49.071975 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec7ae614-bd4b-45a4-862e-1f499ddadbd6-openshift-service-ca\") pod \"maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb\" (UID: \"ec7ae614-bd4b-45a4-862e-1f499ddadbd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb" Oct 30 18:12:49.072140 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:49.072126 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmm92\" (UniqueName: \"kubernetes.io/projected/ec7ae614-bd4b-45a4-862e-1f499ddadbd6-kube-api-access-lmm92\") pod \"maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb\" (UID: \"ec7ae614-bd4b-45a4-862e-1f499ddadbd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb" Oct 30 18:12:49.087702 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:49.087661 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb"] Oct 30 18:12:49.173727 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:49.173688 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec7ae614-bd4b-45a4-862e-1f499ddadbd6-maas-api-tls\") pod \"maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb\" (UID: \"ec7ae614-bd4b-45a4-862e-1f499ddadbd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb" Oct 30 18:12:49.173933 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:49.173770 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec7ae614-bd4b-45a4-862e-1f499ddadbd6-openshift-service-ca\") pod \"maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb\" (UID: \"ec7ae614-bd4b-45a4-862e-1f499ddadbd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb" Oct 30 18:12:49.173933 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:49.173800 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmm92\" (UniqueName: \"kubernetes.io/projected/ec7ae614-bd4b-45a4-862e-1f499ddadbd6-kube-api-access-lmm92\") pod \"maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb\" (UID: \"ec7ae614-bd4b-45a4-862e-1f499ddadbd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb" Oct 30 18:12:49.174067 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:49.173941 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec7ae614-bd4b-45a4-862e-1f499ddadbd6-metrics-certs\") pod \"maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb\" (UID: \"ec7ae614-bd4b-45a4-862e-1f499ddadbd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb" Oct 30 18:12:49.174595 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:49.174567 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec7ae614-bd4b-45a4-862e-1f499ddadbd6-openshift-service-ca\") pod \"maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb\" (UID: \"ec7ae614-bd4b-45a4-862e-1f499ddadbd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb" Oct 30 18:12:49.176560 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:49.176539 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec7ae614-bd4b-45a4-862e-1f499ddadbd6-metrics-certs\") pod \"maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb\" (UID: \"ec7ae614-bd4b-45a4-862e-1f499ddadbd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb" Oct 30 18:12:49.176651 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:49.176633 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec7ae614-bd4b-45a4-862e-1f499ddadbd6-maas-api-tls\") pod \"maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb\" (UID: \"ec7ae614-bd4b-45a4-862e-1f499ddadbd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb" Oct 30 18:12:49.198826 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:49.198788 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmm92\" (UniqueName: \"kubernetes.io/projected/ec7ae614-bd4b-45a4-862e-1f499ddadbd6-kube-api-access-lmm92\") pod \"maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb\" (UID: \"ec7ae614-bd4b-45a4-862e-1f499ddadbd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb" Oct 30 18:12:49.348070 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:49.347965 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb" Oct 30 18:12:49.671195 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:49.671163 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb"] Oct 30 18:12:50.664523 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:50.664482 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb" event={"ID":"ec7ae614-bd4b-45a4-862e-1f499ddadbd6","Type":"ContainerStarted","Data":"a22b5c7bc7ba32aa808919cbaf1a34ca5218eba1e815258de856d22536d67cbb"} Oct 30 18:12:50.664746 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:50.664577 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb" event={"ID":"ec7ae614-bd4b-45a4-862e-1f499ddadbd6","Type":"ContainerStarted","Data":"0107baaad60a579003e56a86cbb784c8c0d467e82acaf6b7ac89c9b267ce0dd1"} Oct 30 18:12:50.664820 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:50.664790 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb" Oct 30 18:12:50.713282 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:50.713219 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb" podStartSLOduration=1.326417907 podStartE2EDuration="1.713204629s" totalImagesPullingTime="386.786722ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:50.709712263 +0000 UTC m=+1223.197349301" watchObservedRunningTime="2026-10-30 18:12:50.713204629 +0000 UTC m=+1223.200841704" Oct 30 18:12:51.627661 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:51.627626 4455 scope.go:122] "RemoveContainer" containerID="199472a03476a3106ef72b622a543763804d52f8cdd439a718fb52cb6810c28a" Oct 30 18:12:52.688058 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:52.688020 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" event={"ID":"9bf6d8f4-3ec1-449f-9399-e2e238d5ee11","Type":"ContainerStarted","Data":"31381ecb3ef748cfd2995d4f1b6bf1bc25b9af849114a2015f3a7e09282afafe"} Oct 30 18:12:52.688535 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:52.688264 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" Oct 30 18:12:56.678200 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:56.678168 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb" Oct 30 18:12:58.703260 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:12:58.703227 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" Oct 30 18:13:00.388511 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:13:00.388472 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:13:10.849151 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:10.849117 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0498d2-5f45b48b8c-vv8th"] Oct 30 18:13:10.855010 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:10.854986 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0498d2-5f45b48b8c-vv8th" Oct 30 18:13:10.879178 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:10.879149 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0498d2-5f45b48b8c-vv8th"] Oct 30 18:13:10.888082 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:10.886709 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa87eb64-d2e5-4448-9058-18eac7330f3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0498d2-5f45b48b8c-vv8th\" (UID: \"aa87eb64-d2e5-4448-9058-18eac7330f3e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0498d2-5f45b48b8c-vv8th" Oct 30 18:13:10.888082 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:10.886967 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvt72\" (UniqueName: \"kubernetes.io/projected/aa87eb64-d2e5-4448-9058-18eac7330f3e-kube-api-access-rvt72\") pod \"payload-pre-processing-e2e-api-b-0498d2-5f45b48b8c-vv8th\" (UID: \"aa87eb64-d2e5-4448-9058-18eac7330f3e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0498d2-5f45b48b8c-vv8th" Oct 30 18:13:10.888082 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:10.887713 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-0498d2\"" Oct 30 18:13:10.983349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:10.955647 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-0498d2-dockercfg-6bdxv\"" Oct 30 18:13:11.032412 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:11.007354 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa87eb64-d2e5-4448-9058-18eac7330f3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0498d2-5f45b48b8c-vv8th\" (UID: \"aa87eb64-d2e5-4448-9058-18eac7330f3e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0498d2-5f45b48b8c-vv8th" Oct 30 18:13:11.032932 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:11.032900 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvt72\" (UniqueName: \"kubernetes.io/projected/aa87eb64-d2e5-4448-9058-18eac7330f3e-kube-api-access-rvt72\") pod \"payload-pre-processing-e2e-api-b-0498d2-5f45b48b8c-vv8th\" (UID: \"aa87eb64-d2e5-4448-9058-18eac7330f3e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0498d2-5f45b48b8c-vv8th" Oct 30 18:13:11.037347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:11.008264 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa87eb64-d2e5-4448-9058-18eac7330f3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0498d2-5f45b48b8c-vv8th\" (UID: \"aa87eb64-d2e5-4448-9058-18eac7330f3e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0498d2-5f45b48b8c-vv8th" Oct 30 18:13:11.082855 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:11.082771 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvt72\" (UniqueName: \"kubernetes.io/projected/aa87eb64-d2e5-4448-9058-18eac7330f3e-kube-api-access-rvt72\") pod \"payload-pre-processing-e2e-api-b-0498d2-5f45b48b8c-vv8th\" (UID: \"aa87eb64-d2e5-4448-9058-18eac7330f3e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0498d2-5f45b48b8c-vv8th" Oct 30 18:13:11.171975 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:11.171363 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0498d2-5f45b48b8c-vv8th" Oct 30 18:13:11.494021 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:13:11.493989 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa87eb64_d2e5_4448_9058_18eac7330f3e.slice/crio-ed7bf6a54968585ca29a3501f64edab19453a6c9514fd2157c9c5c50959dd62c WatchSource:0}: Error finding container ed7bf6a54968585ca29a3501f64edab19453a6c9514fd2157c9c5c50959dd62c: Status 404 returned error can't find the container with id ed7bf6a54968585ca29a3501f64edab19453a6c9514fd2157c9c5c50959dd62c Oct 30 18:13:11.495021 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:11.494995 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0498d2-5f45b48b8c-vv8th"] Oct 30 18:13:11.904640 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:11.904557 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0498d2-5f45b48b8c-vv8th" event={"ID":"aa87eb64-d2e5-4448-9058-18eac7330f3e","Type":"ContainerStarted","Data":"a7db4e09897a5c0488fdd9892e5c0c5487cf522984b191610d358dc6ffe4162c"} Oct 30 18:13:11.905441 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:11.905417 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0498d2-5f45b48b8c-vv8th" event={"ID":"aa87eb64-d2e5-4448-9058-18eac7330f3e","Type":"ContainerStarted","Data":"ed7bf6a54968585ca29a3501f64edab19453a6c9514fd2157c9c5c50959dd62c"} Oct 30 18:13:11.905589 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:11.905578 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0498d2-5f45b48b8c-vv8th" Oct 30 18:13:11.960119 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:11.960059 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0498d2-5f45b48b8c-vv8th" podStartSLOduration=1.96004305 podStartE2EDuration="1.96004305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:11.958178961 +0000 UTC m=+1244.445815995" watchObservedRunningTime="2026-10-30 18:13:11.96004305 +0000 UTC m=+1244.447680084" Oct 30 18:13:22.917115 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:22.917080 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0498d2-5f45b48b8c-vv8th" Oct 30 18:13:23.392586 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:23.392552 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l"] Oct 30 18:13:23.398289 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:23.398269 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l" Oct 30 18:13:23.423190 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:23.420183 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l"] Oct 30 18:13:23.428632 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:23.428604 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-0498d2-serving-cert\"" Oct 30 18:13:23.455263 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:23.455233 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1269fd4d-08e3-4680-8952-1a774748f758-openshift-service-ca\") pod \"maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l\" (UID: \"1269fd4d-08e3-4680-8952-1a774748f758\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l" Oct 30 18:13:23.455436 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:23.455296 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1269fd4d-08e3-4680-8952-1a774748f758-maas-api-tls\") pod \"maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l\" (UID: \"1269fd4d-08e3-4680-8952-1a774748f758\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l" Oct 30 18:13:23.455538 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:23.455521 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1269fd4d-08e3-4680-8952-1a774748f758-metrics-certs\") pod \"maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l\" (UID: \"1269fd4d-08e3-4680-8952-1a774748f758\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l" Oct 30 18:13:23.455584 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:23.455548 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6tn7\" (UniqueName: \"kubernetes.io/projected/1269fd4d-08e3-4680-8952-1a774748f758-kube-api-access-r6tn7\") pod \"maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l\" (UID: \"1269fd4d-08e3-4680-8952-1a774748f758\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l" Oct 30 18:13:23.557581 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:23.557537 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1269fd4d-08e3-4680-8952-1a774748f758-openshift-service-ca\") pod \"maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l\" (UID: \"1269fd4d-08e3-4680-8952-1a774748f758\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l" Oct 30 18:13:23.557780 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:23.557609 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1269fd4d-08e3-4680-8952-1a774748f758-maas-api-tls\") pod \"maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l\" (UID: \"1269fd4d-08e3-4680-8952-1a774748f758\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l" Oct 30 18:13:23.557852 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:13:23.557782 4455 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-0498d2-serving-cert: secret "maas-api-e2e-api-b-0498d2-serving-cert" not found Oct 30 18:13:23.557852 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:23.557829 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1269fd4d-08e3-4680-8952-1a774748f758-metrics-certs\") pod \"maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l\" (UID: \"1269fd4d-08e3-4680-8952-1a774748f758\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l" Oct 30 18:13:23.557956 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:13:23.557859 4455 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1269fd4d-08e3-4680-8952-1a774748f758-maas-api-tls podName:1269fd4d-08e3-4680-8952-1a774748f758 nodeName:}" failed. No retries permitted until 2026-10-30 18:13:24.057839363 +0000 UTC m=+1256.545476411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1269fd4d-08e3-4680-8952-1a774748f758-maas-api-tls") pod "maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l" (UID: "1269fd4d-08e3-4680-8952-1a774748f758") : secret "maas-api-e2e-api-b-0498d2-serving-cert" not found Oct 30 18:13:23.557956 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:23.557900 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6tn7\" (UniqueName: \"kubernetes.io/projected/1269fd4d-08e3-4680-8952-1a774748f758-kube-api-access-r6tn7\") pod \"maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l\" (UID: \"1269fd4d-08e3-4680-8952-1a774748f758\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l" Oct 30 18:13:23.558440 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:23.558414 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1269fd4d-08e3-4680-8952-1a774748f758-openshift-service-ca\") pod \"maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l\" (UID: \"1269fd4d-08e3-4680-8952-1a774748f758\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l" Oct 30 18:13:23.561057 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:23.561035 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1269fd4d-08e3-4680-8952-1a774748f758-metrics-certs\") pod \"maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l\" (UID: \"1269fd4d-08e3-4680-8952-1a774748f758\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l" Oct 30 18:13:23.583630 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:23.583598 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6tn7\" (UniqueName: \"kubernetes.io/projected/1269fd4d-08e3-4680-8952-1a774748f758-kube-api-access-r6tn7\") pod \"maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l\" (UID: \"1269fd4d-08e3-4680-8952-1a774748f758\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l" Oct 30 18:13:24.068088 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:24.068047 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1269fd4d-08e3-4680-8952-1a774748f758-maas-api-tls\") pod \"maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l\" (UID: \"1269fd4d-08e3-4680-8952-1a774748f758\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l" Oct 30 18:13:24.070502 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:24.070475 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1269fd4d-08e3-4680-8952-1a774748f758-maas-api-tls\") pod \"maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l\" (UID: \"1269fd4d-08e3-4680-8952-1a774748f758\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l" Oct 30 18:13:24.316963 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:24.316628 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l" Oct 30 18:13:24.701314 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:24.701276 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l"] Oct 30 18:13:25.045232 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:25.045181 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l" event={"ID":"1269fd4d-08e3-4680-8952-1a774748f758","Type":"ContainerStarted","Data":"6dcad4ba3460d45d3f0eb1fc8242fd39c5fd51e811c04680559f4403fab40903"} Oct 30 18:13:25.845603 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:25.845569 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0bdf03-6d59ffdd86-b25fq"] Oct 30 18:13:25.850453 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:25.850430 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0bdf03-6d59ffdd86-b25fq" Oct 30 18:13:25.883347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:25.880920 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-0bdf03\"" Oct 30 18:13:25.898938 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:25.898584 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-0bdf03-dockercfg-tgcdc\"" Oct 30 18:13:26.039355 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:26.038465 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ad25377-0896-4bba-80a3-97ad09655513-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0bdf03-6d59ffdd86-b25fq\" (UID: \"6ad25377-0896-4bba-80a3-97ad09655513\") " pod="openshift-ingress/payload-processing-e2e-api-b-0bdf03-6d59ffdd86-b25fq" Oct 30 18:13:26.039355 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:26.038743 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5tkp\" (UniqueName: \"kubernetes.io/projected/6ad25377-0896-4bba-80a3-97ad09655513-kube-api-access-r5tkp\") pod \"payload-processing-e2e-api-b-0bdf03-6d59ffdd86-b25fq\" (UID: \"6ad25377-0896-4bba-80a3-97ad09655513\") " pod="openshift-ingress/payload-processing-e2e-api-b-0bdf03-6d59ffdd86-b25fq" Oct 30 18:13:26.056348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:26.055714 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0bdf03-6d59ffdd86-b25fq"] Oct 30 18:13:26.063574 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:26.063499 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l" event={"ID":"1269fd4d-08e3-4680-8952-1a774748f758","Type":"ContainerStarted","Data":"f8f92532432d0e051bc81af96eb141a64354f4d94b205572f323d2db26e3de8e"} Oct 30 18:13:26.065232 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:26.065209 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l" Oct 30 18:13:26.140446 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:26.140365 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ad25377-0896-4bba-80a3-97ad09655513-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0bdf03-6d59ffdd86-b25fq\" (UID: \"6ad25377-0896-4bba-80a3-97ad09655513\") " pod="openshift-ingress/payload-processing-e2e-api-b-0bdf03-6d59ffdd86-b25fq" Oct 30 18:13:26.140784 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:26.140760 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5tkp\" (UniqueName: \"kubernetes.io/projected/6ad25377-0896-4bba-80a3-97ad09655513-kube-api-access-r5tkp\") pod \"payload-processing-e2e-api-b-0bdf03-6d59ffdd86-b25fq\" (UID: \"6ad25377-0896-4bba-80a3-97ad09655513\") " pod="openshift-ingress/payload-processing-e2e-api-b-0bdf03-6d59ffdd86-b25fq" Oct 30 18:13:26.141188 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:26.141163 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ad25377-0896-4bba-80a3-97ad09655513-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0bdf03-6d59ffdd86-b25fq\" (UID: \"6ad25377-0896-4bba-80a3-97ad09655513\") " pod="openshift-ingress/payload-processing-e2e-api-b-0bdf03-6d59ffdd86-b25fq" Oct 30 18:13:26.142985 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:26.142946 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l" podStartSLOduration=2.795644315 podStartE2EDuration="3.14293297s" totalImagesPullingTime="347.288655ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:26.140775956 +0000 UTC m=+1258.628412992" watchObservedRunningTime="2026-10-30 18:13:26.14293297 +0000 UTC m=+1258.630570049" Oct 30 18:13:26.167216 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:26.167176 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5tkp\" (UniqueName: \"kubernetes.io/projected/6ad25377-0896-4bba-80a3-97ad09655513-kube-api-access-r5tkp\") pod \"payload-processing-e2e-api-b-0bdf03-6d59ffdd86-b25fq\" (UID: \"6ad25377-0896-4bba-80a3-97ad09655513\") " pod="openshift-ingress/payload-processing-e2e-api-b-0bdf03-6d59ffdd86-b25fq" Oct 30 18:13:26.185365 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:26.181197 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0bdf03-6d59ffdd86-b25fq" Oct 30 18:13:26.640768 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:13:26.640666 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ad25377_0896_4bba_80a3_97ad09655513.slice/crio-8344de517cdfabe811f315f13c9513cfc448d27287cfb4a443df29559a9b1564 WatchSource:0}: Error finding container 8344de517cdfabe811f315f13c9513cfc448d27287cfb4a443df29559a9b1564: Status 404 returned error can't find the container with id 8344de517cdfabe811f315f13c9513cfc448d27287cfb4a443df29559a9b1564 Oct 30 18:13:26.642451 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:26.642424 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0bdf03-6d59ffdd86-b25fq"] Oct 30 18:13:27.075241 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:27.075202 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0bdf03-6d59ffdd86-b25fq" event={"ID":"6ad25377-0896-4bba-80a3-97ad09655513","Type":"ContainerStarted","Data":"9c5f84440d3df8035fc2f629b49986e6102e6971740e4e93865e1bc35d346a16"} Oct 30 18:13:27.075774 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:27.075302 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0bdf03-6d59ffdd86-b25fq" event={"ID":"6ad25377-0896-4bba-80a3-97ad09655513","Type":"ContainerStarted","Data":"8344de517cdfabe811f315f13c9513cfc448d27287cfb4a443df29559a9b1564"} Oct 30 18:13:27.075774 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:27.075509 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-0bdf03-6d59ffdd86-b25fq" Oct 30 18:13:27.121794 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:27.121743 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-0bdf03-6d59ffdd86-b25fq" podStartSLOduration=2.121728949 podStartE2EDuration="2.121728949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:27.118270066 +0000 UTC m=+1259.605907100" watchObservedRunningTime="2026-10-30 18:13:27.121728949 +0000 UTC m=+1259.609366017" Oct 30 18:13:30.386878 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:13:30.386836 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:13:33.135348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:33.129519 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l" Oct 30 18:13:34.762440 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:34.762401 4455 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:13:34.762988 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:34.762960 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" podUID="d7de539f-17e9-453c-af6b-e83fc7c91239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:36.540348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:36.515007 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:13:36.889551 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:36.889457 4455 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:13:36.889813 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:36.889783 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" podUID="7c9b7598-cec3-43ab-90cf-93e7b407d68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:37.338897 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:37.338452 4455 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:37.548346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:37.545446 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:13:37.569977 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:37.569935 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:13:37.900501 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:37.900469 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:13:38.137350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:38.135535 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-0bdf03-6d59ffdd86-b25fq" Oct 30 18:13:38.138045 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:38.137631 4455 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:13:38.138045 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:38.137683 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" podUID="bbf4bc50-3670-40de-8159-3d8805fdd601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:38.348350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:38.343717 4455 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:13:38.348350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:38.343848 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" podUID="3daa8589-9def-486f-82ee-0c923a18b68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:45.299232 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:45.299181 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0bdf03-fdff56688-vn72p"] Oct 30 18:13:45.330683 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:45.330654 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0bdf03-fdff56688-vn72p" Oct 30 18:13:45.334284 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:45.333969 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0bdf03-fdff56688-vn72p"] Oct 30 18:13:45.354228 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:45.354201 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-0bdf03-serving-cert\"" Oct 30 18:13:45.421815 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:45.421777 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptj72\" (UniqueName: \"kubernetes.io/projected/54820769-b772-419e-932f-30ee9569d49c-kube-api-access-ptj72\") pod \"maas-api-e2e-api-b-0bdf03-fdff56688-vn72p\" (UID: \"54820769-b772-419e-932f-30ee9569d49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0bdf03-fdff56688-vn72p" Oct 30 18:13:45.422021 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:45.421997 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54820769-b772-419e-932f-30ee9569d49c-openshift-service-ca\") pod \"maas-api-e2e-api-b-0bdf03-fdff56688-vn72p\" (UID: \"54820769-b772-419e-932f-30ee9569d49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0bdf03-fdff56688-vn72p" Oct 30 18:13:45.422147 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:45.422121 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54820769-b772-419e-932f-30ee9569d49c-metrics-certs\") pod \"maas-api-e2e-api-b-0bdf03-fdff56688-vn72p\" (UID: \"54820769-b772-419e-932f-30ee9569d49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0bdf03-fdff56688-vn72p" Oct 30 18:13:45.422240 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:45.422224 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54820769-b772-419e-932f-30ee9569d49c-maas-api-tls\") pod \"maas-api-e2e-api-b-0bdf03-fdff56688-vn72p\" (UID: \"54820769-b772-419e-932f-30ee9569d49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0bdf03-fdff56688-vn72p" Oct 30 18:13:45.535637 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:45.531037 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54820769-b772-419e-932f-30ee9569d49c-metrics-certs\") pod \"maas-api-e2e-api-b-0bdf03-fdff56688-vn72p\" (UID: \"54820769-b772-419e-932f-30ee9569d49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0bdf03-fdff56688-vn72p" Oct 30 18:13:45.535637 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:45.531877 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54820769-b772-419e-932f-30ee9569d49c-maas-api-tls\") pod \"maas-api-e2e-api-b-0bdf03-fdff56688-vn72p\" (UID: \"54820769-b772-419e-932f-30ee9569d49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0bdf03-fdff56688-vn72p" Oct 30 18:13:45.535637 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:45.531994 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptj72\" (UniqueName: \"kubernetes.io/projected/54820769-b772-419e-932f-30ee9569d49c-kube-api-access-ptj72\") pod \"maas-api-e2e-api-b-0bdf03-fdff56688-vn72p\" (UID: \"54820769-b772-419e-932f-30ee9569d49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0bdf03-fdff56688-vn72p" Oct 30 18:13:45.535637 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:45.532264 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54820769-b772-419e-932f-30ee9569d49c-openshift-service-ca\") pod \"maas-api-e2e-api-b-0bdf03-fdff56688-vn72p\" (UID: \"54820769-b772-419e-932f-30ee9569d49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0bdf03-fdff56688-vn72p" Oct 30 18:13:45.537620 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:45.537589 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54820769-b772-419e-932f-30ee9569d49c-openshift-service-ca\") pod \"maas-api-e2e-api-b-0bdf03-fdff56688-vn72p\" (UID: \"54820769-b772-419e-932f-30ee9569d49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0bdf03-fdff56688-vn72p" Oct 30 18:13:45.537876 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:13:45.537859 4455 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-0bdf03-serving-cert: secret "maas-api-e2e-api-b-0bdf03-serving-cert" not found Oct 30 18:13:45.548284 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:13:45.548246 4455 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54820769-b772-419e-932f-30ee9569d49c-maas-api-tls podName:54820769-b772-419e-932f-30ee9569d49c nodeName:}" failed. No retries permitted until 2026-10-30 18:13:46.048214838 +0000 UTC m=+1278.535851863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/54820769-b772-419e-932f-30ee9569d49c-maas-api-tls") pod "maas-api-e2e-api-b-0bdf03-fdff56688-vn72p" (UID: "54820769-b772-419e-932f-30ee9569d49c") : secret "maas-api-e2e-api-b-0bdf03-serving-cert" not found Oct 30 18:13:45.584228 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:45.584144 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptj72\" (UniqueName: \"kubernetes.io/projected/54820769-b772-419e-932f-30ee9569d49c-kube-api-access-ptj72\") pod \"maas-api-e2e-api-b-0bdf03-fdff56688-vn72p\" (UID: \"54820769-b772-419e-932f-30ee9569d49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0bdf03-fdff56688-vn72p" Oct 30 18:13:45.585899 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:45.585871 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54820769-b772-419e-932f-30ee9569d49c-metrics-certs\") pod \"maas-api-e2e-api-b-0bdf03-fdff56688-vn72p\" (UID: \"54820769-b772-419e-932f-30ee9569d49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0bdf03-fdff56688-vn72p" Oct 30 18:13:46.056338 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:46.056285 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54820769-b772-419e-932f-30ee9569d49c-maas-api-tls\") pod \"maas-api-e2e-api-b-0bdf03-fdff56688-vn72p\" (UID: \"54820769-b772-419e-932f-30ee9569d49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0bdf03-fdff56688-vn72p" Oct 30 18:13:46.059062 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:46.059031 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54820769-b772-419e-932f-30ee9569d49c-maas-api-tls\") pod \"maas-api-e2e-api-b-0bdf03-fdff56688-vn72p\" (UID: \"54820769-b772-419e-932f-30ee9569d49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0bdf03-fdff56688-vn72p" Oct 30 18:13:46.257959 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:46.257927 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0bdf03-fdff56688-vn72p" Oct 30 18:13:46.584916 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:13:46.584879 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54820769_b772_419e_932f_30ee9569d49c.slice/crio-bf22e39f36efe0ac403e024d843baecc1bcebd5dcfe8e0ef0716c1a70229edac WatchSource:0}: Error finding container bf22e39f36efe0ac403e024d843baecc1bcebd5dcfe8e0ef0716c1a70229edac: Status 404 returned error can't find the container with id bf22e39f36efe0ac403e024d843baecc1bcebd5dcfe8e0ef0716c1a70229edac Oct 30 18:13:46.590717 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:46.590681 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0bdf03-fdff56688-vn72p"] Oct 30 18:13:47.161521 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:47.161488 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0bdf03-fdff56688-vn72p" event={"ID":"54820769-b772-419e-932f-30ee9569d49c","Type":"ContainerStarted","Data":"a4b350b46b77a89af16a793fe10dc4826eb4c918e06a1d167954f9137a7111fb"} Oct 30 18:13:47.162104 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:47.161544 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0bdf03-fdff56688-vn72p" event={"ID":"54820769-b772-419e-932f-30ee9569d49c","Type":"ContainerStarted","Data":"bf22e39f36efe0ac403e024d843baecc1bcebd5dcfe8e0ef0716c1a70229edac"} Oct 30 18:13:47.162104 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:47.161566 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0bdf03-fdff56688-vn72p" Oct 30 18:13:47.210686 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:47.210632 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0bdf03-fdff56688-vn72p" podStartSLOduration=1.816840661 podStartE2EDuration="2.210617194s" totalImagesPullingTime="393.776533ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:47.207540248 +0000 UTC m=+1279.695177283" watchObservedRunningTime="2026-10-30 18:13:47.210617194 +0000 UTC m=+1279.698254229" Oct 30 18:13:53.184504 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:13:53.184474 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0bdf03-fdff56688-vn72p" Oct 30 18:14:00.390992 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:14:00.390949 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:14:07.382858 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:07.382830 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c5891f-7d94c4cd4b-b2w76"] Oct 30 18:14:07.417595 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:07.417564 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c5891f-5586d8fd8c-l2xvx"] Oct 30 18:14:12.384106 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:12.384020 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c5891f-7d94c4cd4b-b2w76" podUID="4b669d0f-5909-4120-87a8-f90ef36cd84b" containerName="payload-processing" containerID="cri-o://9e2e2cefa3175c33b620404054ac1e426f6f30153fb6cb23d3e4d6b7c8456130" gracePeriod=25 Oct 30 18:14:12.418894 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:12.418862 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5891f-5586d8fd8c-l2xvx" podUID="cc8a16fa-fd53-4372-967d-f2f556035841" containerName="payload-pre-processing" containerID="cri-o://471fcb0f6f4881912895dab711929c9488b4770c1676b700f6c6e64c7307649c" gracePeriod=25 Oct 30 18:14:12.738267 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:12.737890 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c5891f-7d94c4cd4b-b2w76" Oct 30 18:14:12.802847 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:12.802817 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b669d0f-5909-4120-87a8-f90ef36cd84b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b669d0f-5909-4120-87a8-f90ef36cd84b-plugins-config-volume\") pod \"4b669d0f-5909-4120-87a8-f90ef36cd84b\" (UID: \"4b669d0f-5909-4120-87a8-f90ef36cd84b\") " Oct 30 18:14:12.803073 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:12.802949 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b669d0f-5909-4120-87a8-f90ef36cd84b-kube-api-access-bmx8p\" (UniqueName: \"kubernetes.io/projected/4b669d0f-5909-4120-87a8-f90ef36cd84b-kube-api-access-bmx8p\") pod \"4b669d0f-5909-4120-87a8-f90ef36cd84b\" (UID: \"4b669d0f-5909-4120-87a8-f90ef36cd84b\") " Oct 30 18:14:12.803345 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:12.803299 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b669d0f-5909-4120-87a8-f90ef36cd84b-plugins-config-volume" pod "4b669d0f-5909-4120-87a8-f90ef36cd84b" (UID: "4b669d0f-5909-4120-87a8-f90ef36cd84b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:14:12.803837 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:12.803817 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b669d0f-5909-4120-87a8-f90ef36cd84b-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:14:12.810165 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:12.810128 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b669d0f-5909-4120-87a8-f90ef36cd84b-kube-api-access-bmx8p" pod "4b669d0f-5909-4120-87a8-f90ef36cd84b" (UID: "4b669d0f-5909-4120-87a8-f90ef36cd84b"). InnerVolumeSpecName "kube-api-access-bmx8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:14:12.813748 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:12.813453 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5891f-5586d8fd8c-l2xvx" Oct 30 18:14:12.904710 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:12.904673 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc8a16fa-fd53-4372-967d-f2f556035841-kube-api-access-j7kkw\" (UniqueName: \"kubernetes.io/projected/cc8a16fa-fd53-4372-967d-f2f556035841-kube-api-access-j7kkw\") pod \"cc8a16fa-fd53-4372-967d-f2f556035841\" (UID: \"cc8a16fa-fd53-4372-967d-f2f556035841\") " Oct 30 18:14:12.904916 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:12.904762 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc8a16fa-fd53-4372-967d-f2f556035841-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc8a16fa-fd53-4372-967d-f2f556035841-plugins-config-volume\") pod \"cc8a16fa-fd53-4372-967d-f2f556035841\" (UID: \"cc8a16fa-fd53-4372-967d-f2f556035841\") " Oct 30 18:14:12.905466 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:12.905440 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc8a16fa-fd53-4372-967d-f2f556035841-plugins-config-volume" pod "cc8a16fa-fd53-4372-967d-f2f556035841" (UID: "cc8a16fa-fd53-4372-967d-f2f556035841"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:14:12.905647 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:12.905474 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmx8p\" (UniqueName: \"kubernetes.io/projected/4b669d0f-5909-4120-87a8-f90ef36cd84b-kube-api-access-bmx8p\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:14:12.907675 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:12.907646 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8a16fa-fd53-4372-967d-f2f556035841-kube-api-access-j7kkw" pod "cc8a16fa-fd53-4372-967d-f2f556035841" (UID: "cc8a16fa-fd53-4372-967d-f2f556035841"). InnerVolumeSpecName "kube-api-access-j7kkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:14:13.007456 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:13.007422 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7kkw\" (UniqueName: \"kubernetes.io/projected/cc8a16fa-fd53-4372-967d-f2f556035841-kube-api-access-j7kkw\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:14:13.007456 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:13.007448 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc8a16fa-fd53-4372-967d-f2f556035841-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:14:13.434617 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:13.434582 4455 generic.go:356] "Generic (PLEG): container finished" podID="4b669d0f-5909-4120-87a8-f90ef36cd84b" containerID="9e2e2cefa3175c33b620404054ac1e426f6f30153fb6cb23d3e4d6b7c8456130" exitCode=0 Oct 30 18:14:13.435062 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:13.434634 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c5891f-7d94c4cd4b-b2w76" Oct 30 18:14:13.435062 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:13.434669 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c5891f-7d94c4cd4b-b2w76" event={"ID":"4b669d0f-5909-4120-87a8-f90ef36cd84b","Type":"ContainerDied","Data":"9e2e2cefa3175c33b620404054ac1e426f6f30153fb6cb23d3e4d6b7c8456130"} Oct 30 18:14:13.435062 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:13.434731 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c5891f-7d94c4cd4b-b2w76" event={"ID":"4b669d0f-5909-4120-87a8-f90ef36cd84b","Type":"ContainerDied","Data":"526fe277e77ad7f27b47dfd0f00e39e9038d8baefb8cb87318c03df2100ad0eb"} Oct 30 18:14:13.435062 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:13.434763 4455 scope.go:122] "RemoveContainer" containerID="9e2e2cefa3175c33b620404054ac1e426f6f30153fb6cb23d3e4d6b7c8456130" Oct 30 18:14:13.436185 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:13.436166 4455 generic.go:356] "Generic (PLEG): container finished" podID="cc8a16fa-fd53-4372-967d-f2f556035841" containerID="471fcb0f6f4881912895dab711929c9488b4770c1676b700f6c6e64c7307649c" exitCode=0 Oct 30 18:14:13.436259 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:13.436227 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5891f-5586d8fd8c-l2xvx" Oct 30 18:14:13.436298 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:13.436260 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5891f-5586d8fd8c-l2xvx" event={"ID":"cc8a16fa-fd53-4372-967d-f2f556035841","Type":"ContainerDied","Data":"471fcb0f6f4881912895dab711929c9488b4770c1676b700f6c6e64c7307649c"} Oct 30 18:14:13.436421 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:13.436314 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5891f-5586d8fd8c-l2xvx" event={"ID":"cc8a16fa-fd53-4372-967d-f2f556035841","Type":"ContainerDied","Data":"2b3280029dc5c9f7fd8c92f78163cef6838375da9fc88d54286f517b005921e7"} Oct 30 18:14:13.447064 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:13.447051 4455 scope.go:122] "RemoveContainer" containerID="9e2e2cefa3175c33b620404054ac1e426f6f30153fb6cb23d3e4d6b7c8456130" Oct 30 18:14:13.447649 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:14:13.447631 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e2e2cefa3175c33b620404054ac1e426f6f30153fb6cb23d3e4d6b7c8456130\": container with ID starting with 9e2e2cefa3175c33b620404054ac1e426f6f30153fb6cb23d3e4d6b7c8456130 not found: ID does not exist" containerID="9e2e2cefa3175c33b620404054ac1e426f6f30153fb6cb23d3e4d6b7c8456130" Oct 30 18:14:13.447723 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:13.447655 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e2e2cefa3175c33b620404054ac1e426f6f30153fb6cb23d3e4d6b7c8456130"} err="failed to get container status \"9e2e2cefa3175c33b620404054ac1e426f6f30153fb6cb23d3e4d6b7c8456130\": rpc error: code = NotFound desc = could not find container \"9e2e2cefa3175c33b620404054ac1e426f6f30153fb6cb23d3e4d6b7c8456130\": container with ID starting with 9e2e2cefa3175c33b620404054ac1e426f6f30153fb6cb23d3e4d6b7c8456130 not found: ID does not exist" Oct 30 18:14:13.447723 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:13.447673 4455 scope.go:122] "RemoveContainer" containerID="471fcb0f6f4881912895dab711929c9488b4770c1676b700f6c6e64c7307649c" Oct 30 18:14:13.459712 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:13.459693 4455 scope.go:122] "RemoveContainer" containerID="471fcb0f6f4881912895dab711929c9488b4770c1676b700f6c6e64c7307649c" Oct 30 18:14:13.460036 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:14:13.460013 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"471fcb0f6f4881912895dab711929c9488b4770c1676b700f6c6e64c7307649c\": container with ID starting with 471fcb0f6f4881912895dab711929c9488b4770c1676b700f6c6e64c7307649c not found: ID does not exist" containerID="471fcb0f6f4881912895dab711929c9488b4770c1676b700f6c6e64c7307649c" Oct 30 18:14:13.460129 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:13.460041 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"471fcb0f6f4881912895dab711929c9488b4770c1676b700f6c6e64c7307649c"} err="failed to get container status \"471fcb0f6f4881912895dab711929c9488b4770c1676b700f6c6e64c7307649c\": rpc error: code = NotFound desc = could not find container \"471fcb0f6f4881912895dab711929c9488b4770c1676b700f6c6e64c7307649c\": container with ID starting with 471fcb0f6f4881912895dab711929c9488b4770c1676b700f6c6e64c7307649c not found: ID does not exist" Oct 30 18:14:13.494074 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:13.494047 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c5891f-5586d8fd8c-l2xvx"] Oct 30 18:14:13.499528 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:13.499500 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c5891f-5586d8fd8c-l2xvx"] Oct 30 18:14:13.534976 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:13.534943 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c5891f-7d94c4cd4b-b2w76"] Oct 30 18:14:13.540869 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:13.540846 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c5891f-7d94c4cd4b-b2w76"] Oct 30 18:14:13.674680 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:13.674605 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b669d0f-5909-4120-87a8-f90ef36cd84b" path="/var/lib/kubelet/pods/4b669d0f-5909-4120-87a8-f90ef36cd84b/volumes" Oct 30 18:14:13.675400 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:13.675186 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc8a16fa-fd53-4372-967d-f2f556035841" path="/var/lib/kubelet/pods/cc8a16fa-fd53-4372-967d-f2f556035841/volumes" Oct 30 18:14:30.390029 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:14:30.389985 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:14:34.532061 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:34.532018 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks"] Oct 30 18:14:34.532595 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:34.532427 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks" podUID="e346787b-4f18-4533-ba53-86c319761600" containerName="maas-api" containerID="cri-o://888a3db4a4b86f701d0675e5eae1984c2ada3f601450f8afb3d2d42507878b6e" gracePeriod=30 Oct 30 18:14:35.227086 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:35.227057 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks" Oct 30 18:14:35.329536 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:35.322799 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e346787b-4f18-4533-ba53-86c319761600-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e346787b-4f18-4533-ba53-86c319761600-metrics-certs\") pod \"e346787b-4f18-4533-ba53-86c319761600\" (UID: \"e346787b-4f18-4533-ba53-86c319761600\") " Oct 30 18:14:35.329536 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:35.322900 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e346787b-4f18-4533-ba53-86c319761600-kube-api-access-4fx2q\" (UniqueName: \"kubernetes.io/projected/e346787b-4f18-4533-ba53-86c319761600-kube-api-access-4fx2q\") pod \"e346787b-4f18-4533-ba53-86c319761600\" (UID: \"e346787b-4f18-4533-ba53-86c319761600\") " Oct 30 18:14:35.329536 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:35.323204 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e346787b-4f18-4533-ba53-86c319761600-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e346787b-4f18-4533-ba53-86c319761600-openshift-service-ca\") pod \"e346787b-4f18-4533-ba53-86c319761600\" (UID: \"e346787b-4f18-4533-ba53-86c319761600\") " Oct 30 18:14:35.329536 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:35.323307 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e346787b-4f18-4533-ba53-86c319761600-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e346787b-4f18-4533-ba53-86c319761600-maas-api-tls\") pod \"e346787b-4f18-4533-ba53-86c319761600\" (UID: \"e346787b-4f18-4533-ba53-86c319761600\") " Oct 30 18:14:35.340574 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:35.340524 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e346787b-4f18-4533-ba53-86c319761600-openshift-service-ca" pod "e346787b-4f18-4533-ba53-86c319761600" (UID: "e346787b-4f18-4533-ba53-86c319761600"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:14:35.341553 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:35.341518 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e346787b-4f18-4533-ba53-86c319761600-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:14:35.345420 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:35.345383 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e346787b-4f18-4533-ba53-86c319761600-maas-api-tls" pod "e346787b-4f18-4533-ba53-86c319761600" (UID: "e346787b-4f18-4533-ba53-86c319761600"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:14:35.346957 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:35.346925 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e346787b-4f18-4533-ba53-86c319761600-kube-api-access-4fx2q" pod "e346787b-4f18-4533-ba53-86c319761600" (UID: "e346787b-4f18-4533-ba53-86c319761600"). InnerVolumeSpecName "kube-api-access-4fx2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:14:35.347069 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:35.347052 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e346787b-4f18-4533-ba53-86c319761600-metrics-certs" pod "e346787b-4f18-4533-ba53-86c319761600" (UID: "e346787b-4f18-4533-ba53-86c319761600"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:14:35.443646 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:35.443551 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e346787b-4f18-4533-ba53-86c319761600-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:14:35.443646 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:35.443585 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e346787b-4f18-4533-ba53-86c319761600-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:14:35.443646 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:35.443595 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fx2q\" (UniqueName: \"kubernetes.io/projected/e346787b-4f18-4533-ba53-86c319761600-kube-api-access-4fx2q\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:14:35.651769 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:35.651642 4455 generic.go:356] "Generic (PLEG): container finished" podID="e346787b-4f18-4533-ba53-86c319761600" containerID="888a3db4a4b86f701d0675e5eae1984c2ada3f601450f8afb3d2d42507878b6e" exitCode=0 Oct 30 18:14:35.651769 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:35.651731 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks" Oct 30 18:14:35.652276 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:35.651745 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks" event={"ID":"e346787b-4f18-4533-ba53-86c319761600","Type":"ContainerDied","Data":"888a3db4a4b86f701d0675e5eae1984c2ada3f601450f8afb3d2d42507878b6e"} Oct 30 18:14:35.652276 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:35.651900 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks" event={"ID":"e346787b-4f18-4533-ba53-86c319761600","Type":"ContainerDied","Data":"0d7e445225eef4cbe6128ef3cfecb3f4d3d6005d4a13007bbd8effaa0b27afaa"} Oct 30 18:14:35.652276 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:35.651949 4455 scope.go:122] "RemoveContainer" containerID="888a3db4a4b86f701d0675e5eae1984c2ada3f601450f8afb3d2d42507878b6e" Oct 30 18:14:35.778865 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:35.777382 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks"] Oct 30 18:14:35.807426 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:35.803534 4455 scope.go:122] "RemoveContainer" containerID="5fb6bdc9436f2012a26ff7c9edf610ec5c9eed20689e530f04469e109516770d" Oct 30 18:14:35.810579 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:35.808517 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c5891f-8798fdcd-5kwks"] Oct 30 18:14:35.913605 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:35.912582 4455 scope.go:122] "RemoveContainer" containerID="888a3db4a4b86f701d0675e5eae1984c2ada3f601450f8afb3d2d42507878b6e" Oct 30 18:14:35.913605 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:14:35.913137 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"888a3db4a4b86f701d0675e5eae1984c2ada3f601450f8afb3d2d42507878b6e\": container with ID starting with 888a3db4a4b86f701d0675e5eae1984c2ada3f601450f8afb3d2d42507878b6e not found: ID does not exist" containerID="888a3db4a4b86f701d0675e5eae1984c2ada3f601450f8afb3d2d42507878b6e" Oct 30 18:14:35.913605 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:35.913174 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"888a3db4a4b86f701d0675e5eae1984c2ada3f601450f8afb3d2d42507878b6e"} err="failed to get container status \"888a3db4a4b86f701d0675e5eae1984c2ada3f601450f8afb3d2d42507878b6e\": rpc error: code = NotFound desc = could not find container \"888a3db4a4b86f701d0675e5eae1984c2ada3f601450f8afb3d2d42507878b6e\": container with ID starting with 888a3db4a4b86f701d0675e5eae1984c2ada3f601450f8afb3d2d42507878b6e not found: ID does not exist" Oct 30 18:14:35.913605 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:35.913221 4455 scope.go:122] "RemoveContainer" containerID="5fb6bdc9436f2012a26ff7c9edf610ec5c9eed20689e530f04469e109516770d" Oct 30 18:14:35.914245 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:14:35.914217 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fb6bdc9436f2012a26ff7c9edf610ec5c9eed20689e530f04469e109516770d\": container with ID starting with 5fb6bdc9436f2012a26ff7c9edf610ec5c9eed20689e530f04469e109516770d not found: ID does not exist" containerID="5fb6bdc9436f2012a26ff7c9edf610ec5c9eed20689e530f04469e109516770d" Oct 30 18:14:35.914369 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:35.914253 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb6bdc9436f2012a26ff7c9edf610ec5c9eed20689e530f04469e109516770d"} err="failed to get container status \"5fb6bdc9436f2012a26ff7c9edf610ec5c9eed20689e530f04469e109516770d\": rpc error: code = NotFound desc = could not find container \"5fb6bdc9436f2012a26ff7c9edf610ec5c9eed20689e530f04469e109516770d\": container with ID starting with 5fb6bdc9436f2012a26ff7c9edf610ec5c9eed20689e530f04469e109516770d not found: ID does not exist" Oct 30 18:14:37.642958 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:37.642923 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e346787b-4f18-4533-ba53-86c319761600" path="/var/lib/kubelet/pods/e346787b-4f18-4533-ba53-86c319761600/volumes" Oct 30 18:14:39.760348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:39.757719 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:39.856493 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:39.856453 4455 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:14:39.856806 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:39.856770 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" podUID="d2c2d808-5274-484e-8520-1f88fa79fc47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:40.780353 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:40.760413 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:40.780353 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:40.762048 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:14:41.125409 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:41.122005 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:14:41.387404 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:41.386066 4455 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:14:41.387404 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:41.386156 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" podUID="1ee94ddb-0834-4ee6-835c-4363b672e1c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:41.387404 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:41.386508 4455 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:14:41.387404 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:41.386548 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" podUID="b933cb99-2e87-4707-a33b-de09b5c03536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:43.152235 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:43.152193 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:43.722861 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:43.722807 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:14:43.723261 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:43.723210 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" podUID="c7c21be6-b9eb-401c-aa3c-af5b62960cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:44.085605 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:44.085473 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:14:44.086670 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:44.086623 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" podUID="4e7b0c73-0041-4549-b47e-76a7aaed2b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:44.160995 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:44.160939 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:14:44.185850 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:44.185807 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:14:44.990829 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:44.990786 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:14:54.647214 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:54.647183 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0498d2-5f45b48b8c-vv8th"] Oct 30 18:14:55.233873 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.233836 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw"] Oct 30 18:14:55.234631 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.234614 4455 cpu_manager.go:414] "containerMap: removing container" podUID="e346787b-4f18-4533-ba53-86c319761600" containerName="maas-api" Oct 30 18:14:55.234690 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.234632 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e346787b-4f18-4533-ba53-86c319761600" containerName="maas-api" Oct 30 18:14:55.234690 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.234667 4455 cpu_manager.go:414] "containerMap: removing container" podUID="cc8a16fa-fd53-4372-967d-f2f556035841" containerName="payload-pre-processing" Oct 30 18:14:55.234690 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.234672 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc8a16fa-fd53-4372-967d-f2f556035841" containerName="payload-pre-processing" Oct 30 18:14:55.234690 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.234686 4455 cpu_manager.go:414] "containerMap: removing container" podUID="4b669d0f-5909-4120-87a8-f90ef36cd84b" containerName="payload-processing" Oct 30 18:14:55.234690 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.234690 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b669d0f-5909-4120-87a8-f90ef36cd84b" containerName="payload-processing" Oct 30 18:14:55.234846 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.234701 4455 cpu_manager.go:414] "containerMap: removing container" podUID="e346787b-4f18-4533-ba53-86c319761600" containerName="maas-api" Oct 30 18:14:55.234846 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.234706 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e346787b-4f18-4533-ba53-86c319761600" containerName="maas-api" Oct 30 18:14:55.234846 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.234803 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="e346787b-4f18-4533-ba53-86c319761600" containerName="maas-api" Oct 30 18:14:55.234846 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.234814 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc8a16fa-fd53-4372-967d-f2f556035841" containerName="payload-pre-processing" Oct 30 18:14:55.234846 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.234821 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b669d0f-5909-4120-87a8-f90ef36cd84b" containerName="payload-processing" Oct 30 18:14:55.234846 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.234827 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="e346787b-4f18-4533-ba53-86c319761600" containerName="maas-api" Oct 30 18:14:55.239089 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.239066 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" Oct 30 18:14:55.288945 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.288903 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-credential-socket\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" Oct 30 18:14:55.289311 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.289288 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-istiod-ca-cert\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" Oct 30 18:14:55.289587 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.289568 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chn62\" (UniqueName: \"kubernetes.io/projected/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-kube-api-access-chn62\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" Oct 30 18:14:55.289816 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.289795 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-istio-data\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" Oct 30 18:14:55.290017 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.289998 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-istio-podinfo\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" Oct 30 18:14:55.290219 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.290202 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-istio-envoy\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" Oct 30 18:14:55.290513 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.290490 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-workload-socket\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" Oct 30 18:14:55.290746 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.290728 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-workload-certs\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" Oct 30 18:14:55.290934 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.290915 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-istio-token\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" Oct 30 18:14:55.351711 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.351652 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-0bdf03-openshift-default-dockercfg-krrwb\"" Oct 30 18:14:55.383550 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.383519 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw"] Oct 30 18:14:55.394438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.394371 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-workload-socket\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" Oct 30 18:14:55.402363 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.394765 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-workload-certs\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" Oct 30 18:14:55.402652 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.402626 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-istio-token\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" Oct 30 18:14:55.402985 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.402963 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-credential-socket\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" Oct 30 18:14:55.403171 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.403154 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-istiod-ca-cert\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" Oct 30 18:14:55.409431 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.409401 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chn62\" (UniqueName: \"kubernetes.io/projected/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-kube-api-access-chn62\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" Oct 30 18:14:55.409687 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.409668 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-istio-data\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" Oct 30 18:14:55.409847 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.409829 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-istio-podinfo\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" Oct 30 18:14:55.420362 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.419459 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-istio-envoy\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" Oct 30 18:14:55.431348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.420559 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-credential-socket\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" Oct 30 18:14:55.431348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.401809 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-workload-certs\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" Oct 30 18:14:55.431348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.421780 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-istiod-ca-cert\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" Oct 30 18:14:55.431348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.422036 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-istio-data\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" Oct 30 18:14:55.431348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.416840 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-workload-socket\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" Oct 30 18:14:55.482083 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.482045 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-istio-envoy\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" Oct 30 18:14:55.482864 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.482835 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-istio-podinfo\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" Oct 30 18:14:55.484103 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.484039 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-istio-token\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" Oct 30 18:14:55.487148 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.487124 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chn62\" (UniqueName: \"kubernetes.io/projected/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-kube-api-access-chn62\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" Oct 30 18:14:55.555807 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.555760 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" Oct 30 18:14:55.877882 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:14:55.877847 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c23fcb1_fecd_48a3_b3da_7dbe9873da88.slice/crio-3c57fac3f6782b2244baad0e4c33cdfec821486495a7499d045c19f62bc60afe WatchSource:0}: Error finding container 3c57fac3f6782b2244baad0e4c33cdfec821486495a7499d045c19f62bc60afe: Status 404 returned error can't find the container with id 3c57fac3f6782b2244baad0e4c33cdfec821486495a7499d045c19f62bc60afe Oct 30 18:14:55.878854 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.878825 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw"] Oct 30 18:14:55.880525 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.880488 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:14:55.880620 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.880598 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:14:55.880689 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:55.880645 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:14:56.182394 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:56.179068 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb"] Oct 30 18:14:56.182394 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:56.179515 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb" podUID="ec7ae614-bd4b-45a4-862e-1f499ddadbd6" containerName="maas-api" containerID="cri-o://a22b5c7bc7ba32aa808919cbaf1a34ca5218eba1e815258de856d22536d67cbb" gracePeriod=30 Oct 30 18:14:56.232065 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:56.230738 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" event={"ID":"1c23fcb1-fecd-48a3-b3da-7dbe9873da88","Type":"ContainerStarted","Data":"4476b58ed44f9c158317daadf6f9f1e9788d8ce768ad0fc90a789a4cc2fe4158"} Oct 30 18:14:56.232065 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:56.230801 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" event={"ID":"1c23fcb1-fecd-48a3-b3da-7dbe9873da88","Type":"ContainerStarted","Data":"3c57fac3f6782b2244baad0e4c33cdfec821486495a7499d045c19f62bc60afe"} Oct 30 18:14:56.300605 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:56.300550 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" podStartSLOduration=1.300535456 podStartE2EDuration="1.300535456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:56.299498403 +0000 UTC m=+1348.787135436" watchObservedRunningTime="2026-10-30 18:14:56.300535456 +0000 UTC m=+1348.788172490" Oct 30 18:14:56.562776 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:56.562695 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" Oct 30 18:14:56.563380 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:56.563037 4455 patch_prober.go:29] interesting pod/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 18:14:56.563380 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:56.563090 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" podUID="1c23fcb1-fecd-48a3-b3da-7dbe9873da88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 18:14:56.952280 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:56.952255 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb" Oct 30 18:14:57.029357 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:57.027927 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec7ae614-bd4b-45a4-862e-1f499ddadbd6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec7ae614-bd4b-45a4-862e-1f499ddadbd6-maas-api-tls\") pod \"ec7ae614-bd4b-45a4-862e-1f499ddadbd6\" (UID: \"ec7ae614-bd4b-45a4-862e-1f499ddadbd6\") " Oct 30 18:14:57.029357 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:57.028063 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec7ae614-bd4b-45a4-862e-1f499ddadbd6-kube-api-access-lmm92\" (UniqueName: \"kubernetes.io/projected/ec7ae614-bd4b-45a4-862e-1f499ddadbd6-kube-api-access-lmm92\") pod \"ec7ae614-bd4b-45a4-862e-1f499ddadbd6\" (UID: \"ec7ae614-bd4b-45a4-862e-1f499ddadbd6\") " Oct 30 18:14:57.029357 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:57.028230 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec7ae614-bd4b-45a4-862e-1f499ddadbd6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec7ae614-bd4b-45a4-862e-1f499ddadbd6-openshift-service-ca\") pod \"ec7ae614-bd4b-45a4-862e-1f499ddadbd6\" (UID: \"ec7ae614-bd4b-45a4-862e-1f499ddadbd6\") " Oct 30 18:14:57.029357 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:57.028532 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec7ae614-bd4b-45a4-862e-1f499ddadbd6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec7ae614-bd4b-45a4-862e-1f499ddadbd6-metrics-certs\") pod \"ec7ae614-bd4b-45a4-862e-1f499ddadbd6\" (UID: \"ec7ae614-bd4b-45a4-862e-1f499ddadbd6\") " Oct 30 18:14:57.030820 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:57.030747 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec7ae614-bd4b-45a4-862e-1f499ddadbd6-openshift-service-ca" pod "ec7ae614-bd4b-45a4-862e-1f499ddadbd6" (UID: "ec7ae614-bd4b-45a4-862e-1f499ddadbd6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:14:57.051680 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:57.051605 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec7ae614-bd4b-45a4-862e-1f499ddadbd6-kube-api-access-lmm92" pod "ec7ae614-bd4b-45a4-862e-1f499ddadbd6" (UID: "ec7ae614-bd4b-45a4-862e-1f499ddadbd6"). InnerVolumeSpecName "kube-api-access-lmm92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:14:57.060508 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:57.060473 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec7ae614-bd4b-45a4-862e-1f499ddadbd6-maas-api-tls" pod "ec7ae614-bd4b-45a4-862e-1f499ddadbd6" (UID: "ec7ae614-bd4b-45a4-862e-1f499ddadbd6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:14:57.092681 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:57.088704 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec7ae614-bd4b-45a4-862e-1f499ddadbd6-metrics-certs" pod "ec7ae614-bd4b-45a4-862e-1f499ddadbd6" (UID: "ec7ae614-bd4b-45a4-862e-1f499ddadbd6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:14:57.132157 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:57.131786 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec7ae614-bd4b-45a4-862e-1f499ddadbd6-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:14:57.132157 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:57.131819 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec7ae614-bd4b-45a4-862e-1f499ddadbd6-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:14:57.132157 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:57.131828 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec7ae614-bd4b-45a4-862e-1f499ddadbd6-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:14:57.132157 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:57.131836 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmm92\" (UniqueName: \"kubernetes.io/projected/ec7ae614-bd4b-45a4-862e-1f499ddadbd6-kube-api-access-lmm92\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:14:57.283350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:57.282598 4455 generic.go:356] "Generic (PLEG): container finished" podID="ec7ae614-bd4b-45a4-862e-1f499ddadbd6" containerID="a22b5c7bc7ba32aa808919cbaf1a34ca5218eba1e815258de856d22536d67cbb" exitCode=0 Oct 30 18:14:57.296001 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:57.294786 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb" Oct 30 18:14:57.313234 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:57.296910 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb" event={"ID":"ec7ae614-bd4b-45a4-862e-1f499ddadbd6","Type":"ContainerDied","Data":"a22b5c7bc7ba32aa808919cbaf1a34ca5218eba1e815258de856d22536d67cbb"} Oct 30 18:14:57.313234 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:57.297031 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb" event={"ID":"ec7ae614-bd4b-45a4-862e-1f499ddadbd6","Type":"ContainerDied","Data":"0107baaad60a579003e56a86cbb784c8c0d467e82acaf6b7ac89c9b267ce0dd1"} Oct 30 18:14:57.313234 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:57.297074 4455 scope.go:122] "RemoveContainer" containerID="a22b5c7bc7ba32aa808919cbaf1a34ca5218eba1e815258de856d22536d67cbb" Oct 30 18:14:57.351040 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:57.351013 4455 scope.go:122] "RemoveContainer" containerID="a22b5c7bc7ba32aa808919cbaf1a34ca5218eba1e815258de856d22536d67cbb" Oct 30 18:14:57.351746 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:14:57.351383 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a22b5c7bc7ba32aa808919cbaf1a34ca5218eba1e815258de856d22536d67cbb\": container with ID starting with a22b5c7bc7ba32aa808919cbaf1a34ca5218eba1e815258de856d22536d67cbb not found: ID does not exist" containerID="a22b5c7bc7ba32aa808919cbaf1a34ca5218eba1e815258de856d22536d67cbb" Oct 30 18:14:57.351746 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:57.351411 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a22b5c7bc7ba32aa808919cbaf1a34ca5218eba1e815258de856d22536d67cbb"} err="failed to get container status \"a22b5c7bc7ba32aa808919cbaf1a34ca5218eba1e815258de856d22536d67cbb\": rpc error: code = NotFound desc = could not find container \"a22b5c7bc7ba32aa808919cbaf1a34ca5218eba1e815258de856d22536d67cbb\": container with ID starting with a22b5c7bc7ba32aa808919cbaf1a34ca5218eba1e815258de856d22536d67cbb not found: ID does not exist" Oct 30 18:14:57.407473 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:57.407435 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb"] Oct 30 18:14:57.409705 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:57.409676 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb"] Oct 30 18:14:57.556882 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:57.556800 4455 patch_prober.go:29] interesting pod/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 18:14:57.556882 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:57.556859 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" podUID="1c23fcb1-fecd-48a3-b3da-7dbe9873da88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 18:14:57.637542 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:57.637504 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec7ae614-bd4b-45a4-862e-1f499ddadbd6" path="/var/lib/kubelet/pods/ec7ae614-bd4b-45a4-862e-1f499ddadbd6/volumes" Oct 30 18:14:58.557153 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:58.557110 4455 patch_prober.go:29] interesting pod/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 18:14:58.557679 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:58.557184 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" podUID="1c23fcb1-fecd-48a3-b3da-7dbe9873da88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 18:14:59.556754 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:59.556718 4455 patch_prober.go:29] interesting pod/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 18:14:59.556942 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:59.556770 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" podUID="1c23fcb1-fecd-48a3-b3da-7dbe9873da88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 18:14:59.647804 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:59.647694 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0498d2-5f45b48b8c-vv8th" podUID="aa87eb64-d2e5-4448-9058-18eac7330f3e" containerName="payload-pre-processing" containerID="cri-o://a7db4e09897a5c0488fdd9892e5c0c5487cf522984b191610d358dc6ffe4162c" gracePeriod=25 Oct 30 18:14:59.688951 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:14:59.688903 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c8e1a-7f8c94686-bpnkb" podUID="ec7ae614-bd4b-45a4-862e-1f499ddadbd6" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.110:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:00.023861 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.023831 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0498d2-5f45b48b8c-vv8th" Oct 30 18:15:00.115362 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.115235 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa87eb64-d2e5-4448-9058-18eac7330f3e-kube-api-access-rvt72\" (UniqueName: \"kubernetes.io/projected/aa87eb64-d2e5-4448-9058-18eac7330f3e-kube-api-access-rvt72\") pod \"aa87eb64-d2e5-4448-9058-18eac7330f3e\" (UID: \"aa87eb64-d2e5-4448-9058-18eac7330f3e\") " Oct 30 18:15:00.115534 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.115370 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa87eb64-d2e5-4448-9058-18eac7330f3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa87eb64-d2e5-4448-9058-18eac7330f3e-plugins-config-volume\") pod \"aa87eb64-d2e5-4448-9058-18eac7330f3e\" (UID: \"aa87eb64-d2e5-4448-9058-18eac7330f3e\") " Oct 30 18:15:00.116250 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.116202 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa87eb64-d2e5-4448-9058-18eac7330f3e-plugins-config-volume" pod "aa87eb64-d2e5-4448-9058-18eac7330f3e" (UID: "aa87eb64-d2e5-4448-9058-18eac7330f3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:00.118835 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.118805 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa87eb64-d2e5-4448-9058-18eac7330f3e-kube-api-access-rvt72" pod "aa87eb64-d2e5-4448-9058-18eac7330f3e" (UID: "aa87eb64-d2e5-4448-9058-18eac7330f3e"). InnerVolumeSpecName "kube-api-access-rvt72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:00.215356 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.214582 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889735-656dq"] Oct 30 18:15:00.215978 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.215953 4455 cpu_manager.go:414] "containerMap: removing container" podUID="ec7ae614-bd4b-45a4-862e-1f499ddadbd6" containerName="maas-api" Oct 30 18:15:00.216127 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.216113 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec7ae614-bd4b-45a4-862e-1f499ddadbd6" containerName="maas-api" Oct 30 18:15:00.216260 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.216249 4455 cpu_manager.go:414] "containerMap: removing container" podUID="e346787b-4f18-4533-ba53-86c319761600" containerName="maas-api" Oct 30 18:15:00.216380 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.216368 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e346787b-4f18-4533-ba53-86c319761600" containerName="maas-api" Oct 30 18:15:00.216486 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.216475 4455 cpu_manager.go:414] "containerMap: removing container" podUID="aa87eb64-d2e5-4448-9058-18eac7330f3e" containerName="payload-pre-processing" Oct 30 18:15:00.216573 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.216560 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa87eb64-d2e5-4448-9058-18eac7330f3e" containerName="payload-pre-processing" Oct 30 18:15:00.216919 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.216867 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec7ae614-bd4b-45a4-862e-1f499ddadbd6" containerName="maas-api" Oct 30 18:15:00.217063 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.217049 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="e346787b-4f18-4533-ba53-86c319761600" containerName="maas-api" Oct 30 18:15:00.217158 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.217148 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa87eb64-d2e5-4448-9058-18eac7330f3e" containerName="payload-pre-processing" Oct 30 18:15:00.221683 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.221667 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-656dq" Oct 30 18:15:00.222724 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.222702 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvt72\" (UniqueName: \"kubernetes.io/projected/aa87eb64-d2e5-4448-9058-18eac7330f3e-kube-api-access-rvt72\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:15:00.222862 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.222847 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa87eb64-d2e5-4448-9058-18eac7330f3e-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:15:00.227531 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.227514 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889735-656dq"] Oct 30 18:15:00.271648 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.271613 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:15:00.324422 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.324394 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba0a949f-05c3-4e1e-85f2-814291a242ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889735-656dq\" (UID: \"ba0a949f-05c3-4e1e-85f2-814291a242ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-656dq" Oct 30 18:15:00.324795 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.324761 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j5jr\" (UniqueName: \"kubernetes.io/projected/ba0a949f-05c3-4e1e-85f2-814291a242ab-kube-api-access-9j5jr\") pod \"maas-api-key-cleanup-29889735-656dq\" (UID: \"ba0a949f-05c3-4e1e-85f2-814291a242ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-656dq" Oct 30 18:15:00.349313 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.349286 4455 generic.go:356] "Generic (PLEG): container finished" podID="aa87eb64-d2e5-4448-9058-18eac7330f3e" containerID="a7db4e09897a5c0488fdd9892e5c0c5487cf522984b191610d358dc6ffe4162c" exitCode=0 Oct 30 18:15:00.349491 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.349364 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0498d2-5f45b48b8c-vv8th" event={"ID":"aa87eb64-d2e5-4448-9058-18eac7330f3e","Type":"ContainerDied","Data":"a7db4e09897a5c0488fdd9892e5c0c5487cf522984b191610d358dc6ffe4162c"} Oct 30 18:15:00.349491 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.349394 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0498d2-5f45b48b8c-vv8th" Oct 30 18:15:00.349491 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.349416 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0498d2-5f45b48b8c-vv8th" event={"ID":"aa87eb64-d2e5-4448-9058-18eac7330f3e","Type":"ContainerDied","Data":"ed7bf6a54968585ca29a3501f64edab19453a6c9514fd2157c9c5c50959dd62c"} Oct 30 18:15:00.349491 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.349459 4455 scope.go:122] "RemoveContainer" containerID="a7db4e09897a5c0488fdd9892e5c0c5487cf522984b191610d358dc6ffe4162c" Oct 30 18:15:00.395545 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.387386 4455 scope.go:122] "RemoveContainer" containerID="a7db4e09897a5c0488fdd9892e5c0c5487cf522984b191610d358dc6ffe4162c" Oct 30 18:15:00.409404 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:15:00.395662 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7db4e09897a5c0488fdd9892e5c0c5487cf522984b191610d358dc6ffe4162c\": container with ID starting with a7db4e09897a5c0488fdd9892e5c0c5487cf522984b191610d358dc6ffe4162c not found: ID does not exist" containerID="a7db4e09897a5c0488fdd9892e5c0c5487cf522984b191610d358dc6ffe4162c" Oct 30 18:15:00.409404 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.395702 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7db4e09897a5c0488fdd9892e5c0c5487cf522984b191610d358dc6ffe4162c"} err="failed to get container status \"a7db4e09897a5c0488fdd9892e5c0c5487cf522984b191610d358dc6ffe4162c\": rpc error: code = NotFound desc = could not find container \"a7db4e09897a5c0488fdd9892e5c0c5487cf522984b191610d358dc6ffe4162c\": container with ID starting with a7db4e09897a5c0488fdd9892e5c0c5487cf522984b191610d358dc6ffe4162c not found: ID does not exist" Oct 30 18:15:00.429390 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:15:00.427494 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:15:00.429390 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.428580 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba0a949f-05c3-4e1e-85f2-814291a242ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889735-656dq\" (UID: \"ba0a949f-05c3-4e1e-85f2-814291a242ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-656dq" Oct 30 18:15:00.429390 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.428787 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j5jr\" (UniqueName: \"kubernetes.io/projected/ba0a949f-05c3-4e1e-85f2-814291a242ab-kube-api-access-9j5jr\") pod \"maas-api-key-cleanup-29889735-656dq\" (UID: \"ba0a949f-05c3-4e1e-85f2-814291a242ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-656dq" Oct 30 18:15:00.430002 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.429977 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba0a949f-05c3-4e1e-85f2-814291a242ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889735-656dq\" (UID: \"ba0a949f-05c3-4e1e-85f2-814291a242ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-656dq" Oct 30 18:15:00.431982 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.431955 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0498d2-5f45b48b8c-vv8th"] Oct 30 18:15:00.437487 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.437464 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0498d2-5f45b48b8c-vv8th"] Oct 30 18:15:00.454161 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.454136 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j5jr\" (UniqueName: \"kubernetes.io/projected/ba0a949f-05c3-4e1e-85f2-814291a242ab-kube-api-access-9j5jr\") pod \"maas-api-key-cleanup-29889735-656dq\" (UID: \"ba0a949f-05c3-4e1e-85f2-814291a242ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-656dq" Oct 30 18:15:00.539915 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.539882 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-656dq" Oct 30 18:15:00.556605 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.556583 4455 patch_prober.go:29] interesting pod/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 18:15:00.556665 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.556634 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" podUID="1c23fcb1-fecd-48a3-b3da-7dbe9873da88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 18:15:00.909431 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:00.909382 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889735-656dq"] Oct 30 18:15:01.359996 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:01.359960 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-656dq" event={"ID":"ba0a949f-05c3-4e1e-85f2-814291a242ab","Type":"ContainerStarted","Data":"f3328651a5e2f87297eb09a8d5f8b18c7f659188d48f329671bd160850d85e10"} Oct 30 18:15:01.557590 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:01.557559 4455 patch_prober.go:29] interesting pod/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 18:15:01.557760 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:01.557623 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" podUID="1c23fcb1-fecd-48a3-b3da-7dbe9873da88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 18:15:01.637213 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:01.637125 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa87eb64-d2e5-4448-9058-18eac7330f3e" path="/var/lib/kubelet/pods/aa87eb64-d2e5-4448-9058-18eac7330f3e/volumes" Oct 30 18:15:02.369996 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:02.369901 4455 generic.go:356] "Generic (PLEG): container finished" podID="ba0a949f-05c3-4e1e-85f2-814291a242ab" containerID="b3f4d87bd838d8a7b99cfd96173063a055a91765a261b5988c71e7ec966d3032" exitCode=60 Oct 30 18:15:02.370463 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:02.370025 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-656dq" event={"ID":"ba0a949f-05c3-4e1e-85f2-814291a242ab","Type":"ContainerDied","Data":"b3f4d87bd838d8a7b99cfd96173063a055a91765a261b5988c71e7ec966d3032"} Oct 30 18:15:02.370524 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:02.370505 4455 scope.go:122] "RemoveContainer" containerID="b3f4d87bd838d8a7b99cfd96173063a055a91765a261b5988c71e7ec966d3032" Oct 30 18:15:02.560826 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:02.560787 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" Oct 30 18:15:02.561294 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:02.561256 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" Oct 30 18:15:02.562174 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:02.562155 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" Oct 30 18:15:03.382096 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:03.382058 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-656dq" event={"ID":"ba0a949f-05c3-4e1e-85f2-814291a242ab","Type":"ContainerStarted","Data":"e0ce90d5cdafecb5874e4b9b27424ed20dce2a9d8978c5b21ce851bf044862f9"} Oct 30 18:15:03.426060 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:03.426011 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-656dq" podStartSLOduration=3.42599857 podStartE2EDuration="3.42599857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:03.424316627 +0000 UTC m=+1355.911953662" watchObservedRunningTime="2026-10-30 18:15:03.42599857 +0000 UTC m=+1355.913635615" Oct 30 18:15:04.392440 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:04.392398 4455 generic.go:356] "Generic (PLEG): container finished" podID="ba0a949f-05c3-4e1e-85f2-814291a242ab" containerID="e0ce90d5cdafecb5874e4b9b27424ed20dce2a9d8978c5b21ce851bf044862f9" exitCode=60 Oct 30 18:15:04.392938 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:04.392544 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-656dq" event={"ID":"ba0a949f-05c3-4e1e-85f2-814291a242ab","Type":"ContainerDied","Data":"e0ce90d5cdafecb5874e4b9b27424ed20dce2a9d8978c5b21ce851bf044862f9"} Oct 30 18:15:04.392938 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:04.392620 4455 scope.go:122] "RemoveContainer" containerID="b3f4d87bd838d8a7b99cfd96173063a055a91765a261b5988c71e7ec966d3032" Oct 30 18:15:04.392938 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:04.392885 4455 scope.go:122] "RemoveContainer" containerID="e0ce90d5cdafecb5874e4b9b27424ed20dce2a9d8978c5b21ce851bf044862f9" Oct 30 18:15:04.393259 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:15:04.393240 4455 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889735-656dq_odh-ai-gateway-infra(ba0a949f-05c3-4e1e-85f2-814291a242ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-656dq" podUID="ba0a949f-05c3-4e1e-85f2-814291a242ab" Oct 30 18:15:05.401894 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:05.401867 4455 scope.go:122] "RemoveContainer" containerID="e0ce90d5cdafecb5874e4b9b27424ed20dce2a9d8978c5b21ce851bf044862f9" Oct 30 18:15:05.402382 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:15:05.402304 4455 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889735-656dq_odh-ai-gateway-infra(ba0a949f-05c3-4e1e-85f2-814291a242ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-656dq" podUID="ba0a949f-05c3-4e1e-85f2-814291a242ab" Oct 30 18:15:07.882979 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:07.882949 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c8f010-6c8445bbdb-fxgzp"] Oct 30 18:15:07.905801 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:07.905767 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c8f010-6c8445bbdb-fxgzp"] Oct 30 18:15:07.905993 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:07.905896 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c8f010-6c8445bbdb-fxgzp" Oct 30 18:15:07.922541 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:07.922515 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c8f010-dockercfg-h2wlb\"" Oct 30 18:15:07.922752 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:07.922615 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c8f010\"" Oct 30 18:15:08.085900 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:08.085866 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqbd5\" (UniqueName: \"kubernetes.io/projected/14a96f10-41c5-44ba-b8ca-86d9fb9520cb-kube-api-access-dqbd5\") pod \"payload-processing-e2e-api-a-c8f010-6c8445bbdb-fxgzp\" (UID: \"14a96f10-41c5-44ba-b8ca-86d9fb9520cb\") " pod="openshift-ingress/payload-processing-e2e-api-a-c8f010-6c8445bbdb-fxgzp" Oct 30 18:15:08.086053 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:08.086001 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14a96f10-41c5-44ba-b8ca-86d9fb9520cb-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c8f010-6c8445bbdb-fxgzp\" (UID: \"14a96f10-41c5-44ba-b8ca-86d9fb9520cb\") " pod="openshift-ingress/payload-processing-e2e-api-a-c8f010-6c8445bbdb-fxgzp" Oct 30 18:15:08.187293 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:08.187210 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14a96f10-41c5-44ba-b8ca-86d9fb9520cb-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c8f010-6c8445bbdb-fxgzp\" (UID: \"14a96f10-41c5-44ba-b8ca-86d9fb9520cb\") " pod="openshift-ingress/payload-processing-e2e-api-a-c8f010-6c8445bbdb-fxgzp" Oct 30 18:15:08.187493 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:08.187479 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqbd5\" (UniqueName: \"kubernetes.io/projected/14a96f10-41c5-44ba-b8ca-86d9fb9520cb-kube-api-access-dqbd5\") pod \"payload-processing-e2e-api-a-c8f010-6c8445bbdb-fxgzp\" (UID: \"14a96f10-41c5-44ba-b8ca-86d9fb9520cb\") " pod="openshift-ingress/payload-processing-e2e-api-a-c8f010-6c8445bbdb-fxgzp" Oct 30 18:15:08.187914 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:08.187893 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14a96f10-41c5-44ba-b8ca-86d9fb9520cb-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c8f010-6c8445bbdb-fxgzp\" (UID: \"14a96f10-41c5-44ba-b8ca-86d9fb9520cb\") " pod="openshift-ingress/payload-processing-e2e-api-a-c8f010-6c8445bbdb-fxgzp" Oct 30 18:15:08.212304 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:08.212269 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqbd5\" (UniqueName: \"kubernetes.io/projected/14a96f10-41c5-44ba-b8ca-86d9fb9520cb-kube-api-access-dqbd5\") pod \"payload-processing-e2e-api-a-c8f010-6c8445bbdb-fxgzp\" (UID: \"14a96f10-41c5-44ba-b8ca-86d9fb9520cb\") " pod="openshift-ingress/payload-processing-e2e-api-a-c8f010-6c8445bbdb-fxgzp" Oct 30 18:15:08.228163 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:08.228134 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c8f010-6c8445bbdb-fxgzp" Oct 30 18:15:08.591355 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:08.587618 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c8f010-6c8445bbdb-fxgzp"] Oct 30 18:15:09.434430 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:09.434396 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c8f010-6c8445bbdb-fxgzp" event={"ID":"14a96f10-41c5-44ba-b8ca-86d9fb9520cb","Type":"ContainerStarted","Data":"60e7740bfe7e0e9997b297611c30cd13b3a131ee5c20319a85933eac680fd8c4"} Oct 30 18:15:09.434908 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:09.434470 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c8f010-6c8445bbdb-fxgzp" event={"ID":"14a96f10-41c5-44ba-b8ca-86d9fb9520cb","Type":"ContainerStarted","Data":"8ecb42d9f6b17cf56050df15e83353cd3ed8ced8dbd38f9c2e30ace4c6b55211"} Oct 30 18:15:09.434908 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:09.434633 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c8f010-6c8445bbdb-fxgzp" Oct 30 18:15:09.479591 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:09.479541 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c8f010-6c8445bbdb-fxgzp" podStartSLOduration=2.47952393 podStartE2EDuration="2.47952393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:09.478791113 +0000 UTC m=+1361.966428148" watchObservedRunningTime="2026-10-30 18:15:09.47952393 +0000 UTC m=+1361.967160964" Oct 30 18:15:15.354832 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:15.354792 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l"] Oct 30 18:15:15.355396 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:15.355067 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l" podUID="1269fd4d-08e3-4680-8952-1a774748f758" containerName="maas-api" containerID="cri-o://f8f92532432d0e051bc81af96eb141a64354f4d94b205572f323d2db26e3de8e" gracePeriod=30 Oct 30 18:15:15.488381 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:15.488349 4455 generic.go:356] "Generic (PLEG): container finished" podID="1269fd4d-08e3-4680-8952-1a774748f758" containerID="f8f92532432d0e051bc81af96eb141a64354f4d94b205572f323d2db26e3de8e" exitCode=0 Oct 30 18:15:15.488490 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:15.488398 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l" event={"ID":"1269fd4d-08e3-4680-8952-1a774748f758","Type":"ContainerDied","Data":"f8f92532432d0e051bc81af96eb141a64354f4d94b205572f323d2db26e3de8e"} Oct 30 18:15:16.074523 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:16.074412 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l" Oct 30 18:15:16.233601 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:16.233569 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1269fd4d-08e3-4680-8952-1a774748f758-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1269fd4d-08e3-4680-8952-1a774748f758-metrics-certs\") pod \"1269fd4d-08e3-4680-8952-1a774748f758\" (UID: \"1269fd4d-08e3-4680-8952-1a774748f758\") " Oct 30 18:15:16.233779 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:16.233679 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1269fd4d-08e3-4680-8952-1a774748f758-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1269fd4d-08e3-4680-8952-1a774748f758-openshift-service-ca\") pod \"1269fd4d-08e3-4680-8952-1a774748f758\" (UID: \"1269fd4d-08e3-4680-8952-1a774748f758\") " Oct 30 18:15:16.233779 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:16.233744 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1269fd4d-08e3-4680-8952-1a774748f758-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1269fd4d-08e3-4680-8952-1a774748f758-maas-api-tls\") pod \"1269fd4d-08e3-4680-8952-1a774748f758\" (UID: \"1269fd4d-08e3-4680-8952-1a774748f758\") " Oct 30 18:15:16.233891 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:16.233874 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1269fd4d-08e3-4680-8952-1a774748f758-kube-api-access-r6tn7\" (UniqueName: \"kubernetes.io/projected/1269fd4d-08e3-4680-8952-1a774748f758-kube-api-access-r6tn7\") pod \"1269fd4d-08e3-4680-8952-1a774748f758\" (UID: \"1269fd4d-08e3-4680-8952-1a774748f758\") " Oct 30 18:15:16.234131 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:16.234094 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1269fd4d-08e3-4680-8952-1a774748f758-openshift-service-ca" pod "1269fd4d-08e3-4680-8952-1a774748f758" (UID: "1269fd4d-08e3-4680-8952-1a774748f758"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:16.234642 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:16.234564 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1269fd4d-08e3-4680-8952-1a774748f758-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:15:16.236004 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:16.235972 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1269fd4d-08e3-4680-8952-1a774748f758-metrics-certs" pod "1269fd4d-08e3-4680-8952-1a774748f758" (UID: "1269fd4d-08e3-4680-8952-1a774748f758"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:16.236004 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:16.235993 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1269fd4d-08e3-4680-8952-1a774748f758-maas-api-tls" pod "1269fd4d-08e3-4680-8952-1a774748f758" (UID: "1269fd4d-08e3-4680-8952-1a774748f758"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:16.236201 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:16.236180 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1269fd4d-08e3-4680-8952-1a774748f758-kube-api-access-r6tn7" pod "1269fd4d-08e3-4680-8952-1a774748f758" (UID: "1269fd4d-08e3-4680-8952-1a774748f758"). InnerVolumeSpecName "kube-api-access-r6tn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:16.336182 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:16.336086 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1269fd4d-08e3-4680-8952-1a774748f758-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:15:16.336182 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:16.336127 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1269fd4d-08e3-4680-8952-1a774748f758-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:15:16.336182 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:16.336144 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6tn7\" (UniqueName: \"kubernetes.io/projected/1269fd4d-08e3-4680-8952-1a774748f758-kube-api-access-r6tn7\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:15:16.497061 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:16.497029 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l" Oct 30 18:15:16.497629 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:16.497065 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l" event={"ID":"1269fd4d-08e3-4680-8952-1a774748f758","Type":"ContainerDied","Data":"6dcad4ba3460d45d3f0eb1fc8242fd39c5fd51e811c04680559f4403fab40903"} Oct 30 18:15:16.497629 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:16.497162 4455 scope.go:122] "RemoveContainer" containerID="f8f92532432d0e051bc81af96eb141a64354f4d94b205572f323d2db26e3de8e" Oct 30 18:15:16.558291 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:16.558261 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l"] Oct 30 18:15:16.562574 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:16.562550 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0498d2-556c6d8bc4-hdj8l"] Oct 30 18:15:17.635627 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:17.635595 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1269fd4d-08e3-4680-8952-1a774748f758" path="/var/lib/kubelet/pods/1269fd4d-08e3-4680-8952-1a774748f758/volumes" Oct 30 18:15:17.636777 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:17.636759 4455 scope.go:122] "RemoveContainer" containerID="e0ce90d5cdafecb5874e4b9b27424ed20dce2a9d8978c5b21ce851bf044862f9" Oct 30 18:15:18.514896 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:18.514855 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-656dq" event={"ID":"ba0a949f-05c3-4e1e-85f2-814291a242ab","Type":"ContainerStarted","Data":"83525a413166cb9f2d8127ebf2255ec9ef1ad0800427e3093e92233fe4ed8713"} Oct 30 18:15:19.526640 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:19.526603 4455 generic.go:356] "Generic (PLEG): container finished" podID="ba0a949f-05c3-4e1e-85f2-814291a242ab" containerID="83525a413166cb9f2d8127ebf2255ec9ef1ad0800427e3093e92233fe4ed8713" exitCode=60 Oct 30 18:15:19.527148 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:19.526689 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-656dq" event={"ID":"ba0a949f-05c3-4e1e-85f2-814291a242ab","Type":"ContainerDied","Data":"83525a413166cb9f2d8127ebf2255ec9ef1ad0800427e3093e92233fe4ed8713"} Oct 30 18:15:19.527148 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:19.526748 4455 scope.go:122] "RemoveContainer" containerID="e0ce90d5cdafecb5874e4b9b27424ed20dce2a9d8978c5b21ce851bf044862f9" Oct 30 18:15:19.527295 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:19.527227 4455 scope.go:122] "RemoveContainer" containerID="83525a413166cb9f2d8127ebf2255ec9ef1ad0800427e3093e92233fe4ed8713" Oct 30 18:15:19.527671 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:15:19.527639 4455 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889735-656dq_odh-ai-gateway-infra(ba0a949f-05c3-4e1e-85f2-814291a242ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-656dq" podUID="ba0a949f-05c3-4e1e-85f2-814291a242ab" Oct 30 18:15:19.590207 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:19.590175 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889735-656dq"] Oct 30 18:15:19.874857 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:19.874774 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp"] Oct 30 18:15:19.875644 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:19.875627 4455 cpu_manager.go:414] "containerMap: removing container" podUID="1269fd4d-08e3-4680-8952-1a774748f758" containerName="maas-api" Oct 30 18:15:19.875716 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:19.875645 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1269fd4d-08e3-4680-8952-1a774748f758" containerName="maas-api" Oct 30 18:15:19.875779 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:19.875770 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="1269fd4d-08e3-4680-8952-1a774748f758" containerName="maas-api" Oct 30 18:15:19.880804 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:19.880774 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp" Oct 30 18:15:19.904708 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:19.904666 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp"] Oct 30 18:15:19.910028 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:19.909999 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c8f010-serving-cert\"" Oct 30 18:15:19.993652 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:19.993616 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52e2e782-aab7-463d-b91a-e37a774091ab-maas-api-tls\") pod \"maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp\" (UID: \"52e2e782-aab7-463d-b91a-e37a774091ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp" Oct 30 18:15:19.993836 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:19.993741 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52e2e782-aab7-463d-b91a-e37a774091ab-openshift-service-ca\") pod \"maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp\" (UID: \"52e2e782-aab7-463d-b91a-e37a774091ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp" Oct 30 18:15:19.993836 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:19.993783 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csk8q\" (UniqueName: \"kubernetes.io/projected/52e2e782-aab7-463d-b91a-e37a774091ab-kube-api-access-csk8q\") pod \"maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp\" (UID: \"52e2e782-aab7-463d-b91a-e37a774091ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp" Oct 30 18:15:19.994000 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:19.993987 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52e2e782-aab7-463d-b91a-e37a774091ab-metrics-certs\") pod \"maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp\" (UID: \"52e2e782-aab7-463d-b91a-e37a774091ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp" Oct 30 18:15:20.095307 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:20.095271 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52e2e782-aab7-463d-b91a-e37a774091ab-openshift-service-ca\") pod \"maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp\" (UID: \"52e2e782-aab7-463d-b91a-e37a774091ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp" Oct 30 18:15:20.095493 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:20.095339 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csk8q\" (UniqueName: \"kubernetes.io/projected/52e2e782-aab7-463d-b91a-e37a774091ab-kube-api-access-csk8q\") pod \"maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp\" (UID: \"52e2e782-aab7-463d-b91a-e37a774091ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp" Oct 30 18:15:20.095552 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:20.095532 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52e2e782-aab7-463d-b91a-e37a774091ab-metrics-certs\") pod \"maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp\" (UID: \"52e2e782-aab7-463d-b91a-e37a774091ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp" Oct 30 18:15:20.095728 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:20.095712 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52e2e782-aab7-463d-b91a-e37a774091ab-maas-api-tls\") pod \"maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp\" (UID: \"52e2e782-aab7-463d-b91a-e37a774091ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp" Oct 30 18:15:20.096055 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:20.096035 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52e2e782-aab7-463d-b91a-e37a774091ab-openshift-service-ca\") pod \"maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp\" (UID: \"52e2e782-aab7-463d-b91a-e37a774091ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp" Oct 30 18:15:20.098202 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:20.098177 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52e2e782-aab7-463d-b91a-e37a774091ab-metrics-certs\") pod \"maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp\" (UID: \"52e2e782-aab7-463d-b91a-e37a774091ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp" Oct 30 18:15:20.098302 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:20.098223 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52e2e782-aab7-463d-b91a-e37a774091ab-maas-api-tls\") pod \"maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp\" (UID: \"52e2e782-aab7-463d-b91a-e37a774091ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp" Oct 30 18:15:20.120458 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:20.120426 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csk8q\" (UniqueName: \"kubernetes.io/projected/52e2e782-aab7-463d-b91a-e37a774091ab-kube-api-access-csk8q\") pod \"maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp\" (UID: \"52e2e782-aab7-463d-b91a-e37a774091ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp" Oct 30 18:15:20.197764 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:20.197663 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp" Oct 30 18:15:20.452000 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:20.451922 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c8f010-6c8445bbdb-fxgzp" Oct 30 18:15:20.574865 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:15:20.574829 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52e2e782_aab7_463d_b91a_e37a774091ab.slice/crio-58f2aecc69b8a8cf840969989f262437e137e1a056f32ce74058e8edd8c24535 WatchSource:0}: Error finding container 58f2aecc69b8a8cf840969989f262437e137e1a056f32ce74058e8edd8c24535: Status 404 returned error can't find the container with id 58f2aecc69b8a8cf840969989f262437e137e1a056f32ce74058e8edd8c24535 Oct 30 18:15:20.577755 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:20.577725 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp"] Oct 30 18:15:20.774347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:20.761921 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-656dq" Oct 30 18:15:20.922355 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:20.922303 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba0a949f-05c3-4e1e-85f2-814291a242ab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba0a949f-05c3-4e1e-85f2-814291a242ab-maas-api-service-ca\") pod \"ba0a949f-05c3-4e1e-85f2-814291a242ab\" (UID: \"ba0a949f-05c3-4e1e-85f2-814291a242ab\") " Oct 30 18:15:20.922554 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:20.922376 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba0a949f-05c3-4e1e-85f2-814291a242ab-kube-api-access-9j5jr\" (UniqueName: \"kubernetes.io/projected/ba0a949f-05c3-4e1e-85f2-814291a242ab-kube-api-access-9j5jr\") pod \"ba0a949f-05c3-4e1e-85f2-814291a242ab\" (UID: \"ba0a949f-05c3-4e1e-85f2-814291a242ab\") " Oct 30 18:15:20.922679 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:20.922647 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba0a949f-05c3-4e1e-85f2-814291a242ab-maas-api-service-ca" pod "ba0a949f-05c3-4e1e-85f2-814291a242ab" (UID: "ba0a949f-05c3-4e1e-85f2-814291a242ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:20.923001 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:20.922983 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba0a949f-05c3-4e1e-85f2-814291a242ab-maas-api-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:15:20.924832 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:20.924809 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba0a949f-05c3-4e1e-85f2-814291a242ab-kube-api-access-9j5jr" pod "ba0a949f-05c3-4e1e-85f2-814291a242ab" (UID: "ba0a949f-05c3-4e1e-85f2-814291a242ab"). InnerVolumeSpecName "kube-api-access-9j5jr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:21.024271 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:21.024244 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j5jr\" (UniqueName: \"kubernetes.io/projected/ba0a949f-05c3-4e1e-85f2-814291a242ab-kube-api-access-9j5jr\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:15:21.551272 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:21.551240 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-656dq" Oct 30 18:15:21.551499 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:21.551272 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-656dq" event={"ID":"ba0a949f-05c3-4e1e-85f2-814291a242ab","Type":"ContainerDied","Data":"f3328651a5e2f87297eb09a8d5f8b18c7f659188d48f329671bd160850d85e10"} Oct 30 18:15:21.551499 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:21.551381 4455 scope.go:122] "RemoveContainer" containerID="83525a413166cb9f2d8127ebf2255ec9ef1ad0800427e3093e92233fe4ed8713" Oct 30 18:15:21.553243 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:21.553211 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp" event={"ID":"52e2e782-aab7-463d-b91a-e37a774091ab","Type":"ContainerStarted","Data":"d114d95ea33ea3e2f4eff1939e0452977bd1ddeed14b00411742c7a8478297b7"} Oct 30 18:15:21.553408 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:21.553253 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp" event={"ID":"52e2e782-aab7-463d-b91a-e37a774091ab","Type":"ContainerStarted","Data":"58f2aecc69b8a8cf840969989f262437e137e1a056f32ce74058e8edd8c24535"} Oct 30 18:15:21.553408 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:21.553402 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp" Oct 30 18:15:21.614367 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:21.614266 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp" podStartSLOduration=2.167867617 podStartE2EDuration="2.614249446s" totalImagesPullingTime="446.381829ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:21.613595404 +0000 UTC m=+1374.101232439" watchObservedRunningTime="2026-10-30 18:15:21.614249446 +0000 UTC m=+1374.101886481" Oct 30 18:15:21.675118 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:21.675079 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889735-656dq"] Oct 30 18:15:21.680212 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:21.680184 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889735-656dq"] Oct 30 18:15:23.644778 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:23.644734 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba0a949f-05c3-4e1e-85f2-814291a242ab" path="/var/lib/kubelet/pods/ba0a949f-05c3-4e1e-85f2-814291a242ab/volumes" Oct 30 18:15:25.685117 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:25.685083 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0bdf03-6d59ffdd86-b25fq"] Oct 30 18:15:27.569781 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:27.569753 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp" Oct 30 18:15:30.391066 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:15:30.391030 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:15:30.685537 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:30.685419 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-0bdf03-6d59ffdd86-b25fq" podUID="6ad25377-0896-4bba-80a3-97ad09655513" containerName="payload-processing" containerID="cri-o://9c5f84440d3df8035fc2f629b49986e6102e6971740e4e93865e1bc35d346a16" gracePeriod=25 Oct 30 18:15:30.998957 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:30.998920 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0bdf03-6d59ffdd86-b25fq" Oct 30 18:15:31.059009 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:31.058981 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ad25377-0896-4bba-80a3-97ad09655513-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ad25377-0896-4bba-80a3-97ad09655513-plugins-config-volume\") pod \"6ad25377-0896-4bba-80a3-97ad09655513\" (UID: \"6ad25377-0896-4bba-80a3-97ad09655513\") " Oct 30 18:15:31.059214 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:31.059035 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ad25377-0896-4bba-80a3-97ad09655513-kube-api-access-r5tkp\" (UniqueName: \"kubernetes.io/projected/6ad25377-0896-4bba-80a3-97ad09655513-kube-api-access-r5tkp\") pod \"6ad25377-0896-4bba-80a3-97ad09655513\" (UID: \"6ad25377-0896-4bba-80a3-97ad09655513\") " Oct 30 18:15:31.059480 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:31.059447 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ad25377-0896-4bba-80a3-97ad09655513-plugins-config-volume" pod "6ad25377-0896-4bba-80a3-97ad09655513" (UID: "6ad25377-0896-4bba-80a3-97ad09655513"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:31.059826 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:31.059808 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ad25377-0896-4bba-80a3-97ad09655513-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:15:31.061540 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:31.061509 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad25377-0896-4bba-80a3-97ad09655513-kube-api-access-r5tkp" pod "6ad25377-0896-4bba-80a3-97ad09655513" (UID: "6ad25377-0896-4bba-80a3-97ad09655513"). InnerVolumeSpecName "kube-api-access-r5tkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:31.161758 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:31.161717 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5tkp\" (UniqueName: \"kubernetes.io/projected/6ad25377-0896-4bba-80a3-97ad09655513-kube-api-access-r5tkp\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:15:31.655572 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:31.655544 4455 generic.go:356] "Generic (PLEG): container finished" podID="6ad25377-0896-4bba-80a3-97ad09655513" containerID="9c5f84440d3df8035fc2f629b49986e6102e6971740e4e93865e1bc35d346a16" exitCode=0 Oct 30 18:15:31.655940 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:31.655587 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0bdf03-6d59ffdd86-b25fq" Oct 30 18:15:31.655940 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:31.655618 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0bdf03-6d59ffdd86-b25fq" event={"ID":"6ad25377-0896-4bba-80a3-97ad09655513","Type":"ContainerDied","Data":"9c5f84440d3df8035fc2f629b49986e6102e6971740e4e93865e1bc35d346a16"} Oct 30 18:15:31.655940 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:31.655674 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0bdf03-6d59ffdd86-b25fq" event={"ID":"6ad25377-0896-4bba-80a3-97ad09655513","Type":"ContainerDied","Data":"8344de517cdfabe811f315f13c9513cfc448d27287cfb4a443df29559a9b1564"} Oct 30 18:15:31.655940 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:31.655696 4455 scope.go:122] "RemoveContainer" containerID="9c5f84440d3df8035fc2f629b49986e6102e6971740e4e93865e1bc35d346a16" Oct 30 18:15:31.666289 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:31.666268 4455 scope.go:122] "RemoveContainer" containerID="9c5f84440d3df8035fc2f629b49986e6102e6971740e4e93865e1bc35d346a16" Oct 30 18:15:31.666662 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:15:31.666638 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c5f84440d3df8035fc2f629b49986e6102e6971740e4e93865e1bc35d346a16\": container with ID starting with 9c5f84440d3df8035fc2f629b49986e6102e6971740e4e93865e1bc35d346a16 not found: ID does not exist" containerID="9c5f84440d3df8035fc2f629b49986e6102e6971740e4e93865e1bc35d346a16" Oct 30 18:15:31.666757 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:31.666684 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c5f84440d3df8035fc2f629b49986e6102e6971740e4e93865e1bc35d346a16"} err="failed to get container status \"9c5f84440d3df8035fc2f629b49986e6102e6971740e4e93865e1bc35d346a16\": rpc error: code = NotFound desc = could not find container \"9c5f84440d3df8035fc2f629b49986e6102e6971740e4e93865e1bc35d346a16\": container with ID starting with 9c5f84440d3df8035fc2f629b49986e6102e6971740e4e93865e1bc35d346a16 not found: ID does not exist" Oct 30 18:15:31.713771 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:31.713741 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0bdf03-6d59ffdd86-b25fq"] Oct 30 18:15:31.723150 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:31.723121 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0bdf03-6d59ffdd86-b25fq"] Oct 30 18:15:33.677392 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:33.674122 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ad25377-0896-4bba-80a3-97ad09655513" path="/var/lib/kubelet/pods/6ad25377-0896-4bba-80a3-97ad09655513/volumes" Oct 30 18:15:48.447612 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:48.447524 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0bdf03-fdff56688-vn72p"] Oct 30 18:15:48.448091 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:48.447847 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0bdf03-fdff56688-vn72p" podUID="54820769-b772-419e-932f-30ee9569d49c" containerName="maas-api" containerID="cri-o://a4b350b46b77a89af16a793fe10dc4826eb4c918e06a1d167954f9137a7111fb" gracePeriod=30 Oct 30 18:15:49.183745 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:49.178562 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0bdf03-fdff56688-vn72p" Oct 30 18:15:49.185762 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:49.185738 4455 generic.go:356] "Generic (PLEG): container finished" podID="54820769-b772-419e-932f-30ee9569d49c" containerID="a4b350b46b77a89af16a793fe10dc4826eb4c918e06a1d167954f9137a7111fb" exitCode=0 Oct 30 18:15:49.186609 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:49.185922 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0bdf03-fdff56688-vn72p" event={"ID":"54820769-b772-419e-932f-30ee9569d49c","Type":"ContainerDied","Data":"a4b350b46b77a89af16a793fe10dc4826eb4c918e06a1d167954f9137a7111fb"} Oct 30 18:15:49.186739 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:49.186681 4455 scope.go:122] "RemoveContainer" containerID="a4b350b46b77a89af16a793fe10dc4826eb4c918e06a1d167954f9137a7111fb" Oct 30 18:15:49.302347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:49.302226 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54820769-b772-419e-932f-30ee9569d49c-kube-api-access-ptj72\" (UniqueName: \"kubernetes.io/projected/54820769-b772-419e-932f-30ee9569d49c-kube-api-access-ptj72\") pod \"54820769-b772-419e-932f-30ee9569d49c\" (UID: \"54820769-b772-419e-932f-30ee9569d49c\") " Oct 30 18:15:49.302347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:49.302311 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54820769-b772-419e-932f-30ee9569d49c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/54820769-b772-419e-932f-30ee9569d49c-metrics-certs\") pod \"54820769-b772-419e-932f-30ee9569d49c\" (UID: \"54820769-b772-419e-932f-30ee9569d49c\") " Oct 30 18:15:49.302604 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:49.302397 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54820769-b772-419e-932f-30ee9569d49c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54820769-b772-419e-932f-30ee9569d49c-maas-api-tls\") pod \"54820769-b772-419e-932f-30ee9569d49c\" (UID: \"54820769-b772-419e-932f-30ee9569d49c\") " Oct 30 18:15:49.302847 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:49.302829 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54820769-b772-419e-932f-30ee9569d49c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54820769-b772-419e-932f-30ee9569d49c-openshift-service-ca\") pod \"54820769-b772-419e-932f-30ee9569d49c\" (UID: \"54820769-b772-419e-932f-30ee9569d49c\") " Oct 30 18:15:49.303294 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:49.303269 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54820769-b772-419e-932f-30ee9569d49c-openshift-service-ca" pod "54820769-b772-419e-932f-30ee9569d49c" (UID: "54820769-b772-419e-932f-30ee9569d49c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:49.304484 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:49.304462 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54820769-b772-419e-932f-30ee9569d49c-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:15:49.307980 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:49.307953 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54820769-b772-419e-932f-30ee9569d49c-kube-api-access-ptj72" pod "54820769-b772-419e-932f-30ee9569d49c" (UID: "54820769-b772-419e-932f-30ee9569d49c"). InnerVolumeSpecName "kube-api-access-ptj72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:49.308396 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:49.308371 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54820769-b772-419e-932f-30ee9569d49c-metrics-certs" pod "54820769-b772-419e-932f-30ee9569d49c" (UID: "54820769-b772-419e-932f-30ee9569d49c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:49.309568 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:49.309541 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54820769-b772-419e-932f-30ee9569d49c-maas-api-tls" pod "54820769-b772-419e-932f-30ee9569d49c" (UID: "54820769-b772-419e-932f-30ee9569d49c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:49.421346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:49.413730 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54820769-b772-419e-932f-30ee9569d49c-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:15:49.421346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:49.413771 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptj72\" (UniqueName: \"kubernetes.io/projected/54820769-b772-419e-932f-30ee9569d49c-kube-api-access-ptj72\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:15:49.421346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:49.413790 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54820769-b772-419e-932f-30ee9569d49c-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:15:50.195152 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:50.195117 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0bdf03-fdff56688-vn72p" Oct 30 18:15:50.195152 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:50.195138 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0bdf03-fdff56688-vn72p" event={"ID":"54820769-b772-419e-932f-30ee9569d49c","Type":"ContainerDied","Data":"bf22e39f36efe0ac403e024d843baecc1bcebd5dcfe8e0ef0716c1a70229edac"} Oct 30 18:15:50.253114 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:50.253085 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0bdf03-fdff56688-vn72p"] Oct 30 18:15:50.266562 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:50.266530 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0bdf03-fdff56688-vn72p"] Oct 30 18:15:51.635664 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:51.635633 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54820769-b772-419e-932f-30ee9569d49c" path="/var/lib/kubelet/pods/54820769-b772-419e-932f-30ee9569d49c/volumes" Oct 30 18:15:53.373899 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.373864 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-da151d-6cfcff59d7-rqzfd"] Oct 30 18:15:53.374634 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.374617 4455 cpu_manager.go:414] "containerMap: removing container" podUID="ba0a949f-05c3-4e1e-85f2-814291a242ab" containerName="cleanup" Oct 30 18:15:53.374701 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.374636 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba0a949f-05c3-4e1e-85f2-814291a242ab" containerName="cleanup" Oct 30 18:15:53.374701 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.374663 4455 cpu_manager.go:414] "containerMap: removing container" podUID="ba0a949f-05c3-4e1e-85f2-814291a242ab" containerName="cleanup" Oct 30 18:15:53.374701 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.374668 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba0a949f-05c3-4e1e-85f2-814291a242ab" containerName="cleanup" Oct 30 18:15:53.374701 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.374676 4455 cpu_manager.go:414] "containerMap: removing container" podUID="6ad25377-0896-4bba-80a3-97ad09655513" containerName="payload-processing" Oct 30 18:15:53.374701 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.374680 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ad25377-0896-4bba-80a3-97ad09655513" containerName="payload-processing" Oct 30 18:15:53.374701 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.374687 4455 cpu_manager.go:414] "containerMap: removing container" podUID="54820769-b772-419e-932f-30ee9569d49c" containerName="maas-api" Oct 30 18:15:53.374701 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.374691 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54820769-b772-419e-932f-30ee9569d49c" containerName="maas-api" Oct 30 18:15:53.374921 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.374810 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba0a949f-05c3-4e1e-85f2-814291a242ab" containerName="cleanup" Oct 30 18:15:53.374921 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.374822 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba0a949f-05c3-4e1e-85f2-814291a242ab" containerName="cleanup" Oct 30 18:15:53.374921 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.374832 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ad25377-0896-4bba-80a3-97ad09655513" containerName="payload-processing" Oct 30 18:15:53.374921 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.374838 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="54820769-b772-419e-932f-30ee9569d49c" containerName="maas-api" Oct 30 18:15:53.380594 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.380575 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da151d-6cfcff59d7-rqzfd" Oct 30 18:15:53.403133 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.403101 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-da151d-6cfcff59d7-rqzfd"] Oct 30 18:15:53.413367 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.408633 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-da151d\"" Oct 30 18:15:53.413367 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.412439 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-da151d-dockercfg-tzqjj\"" Oct 30 18:15:53.538351 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.532238 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n472l\" (UniqueName: \"kubernetes.io/projected/ebc64c0e-74bd-4130-bf28-65c0e73f2197-kube-api-access-n472l\") pod \"payload-pre-processing-e2e-api-b-da151d-6cfcff59d7-rqzfd\" (UID: \"ebc64c0e-74bd-4130-bf28-65c0e73f2197\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-da151d-6cfcff59d7-rqzfd" Oct 30 18:15:53.538351 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.532441 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc64c0e-74bd-4130-bf28-65c0e73f2197-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-da151d-6cfcff59d7-rqzfd\" (UID: \"ebc64c0e-74bd-4130-bf28-65c0e73f2197\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-da151d-6cfcff59d7-rqzfd" Oct 30 18:15:53.643745 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.643646 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc64c0e-74bd-4130-bf28-65c0e73f2197-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-da151d-6cfcff59d7-rqzfd\" (UID: \"ebc64c0e-74bd-4130-bf28-65c0e73f2197\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-da151d-6cfcff59d7-rqzfd" Oct 30 18:15:53.644228 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.644203 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n472l\" (UniqueName: \"kubernetes.io/projected/ebc64c0e-74bd-4130-bf28-65c0e73f2197-kube-api-access-n472l\") pod \"payload-pre-processing-e2e-api-b-da151d-6cfcff59d7-rqzfd\" (UID: \"ebc64c0e-74bd-4130-bf28-65c0e73f2197\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-da151d-6cfcff59d7-rqzfd" Oct 30 18:15:53.645972 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.645948 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc64c0e-74bd-4130-bf28-65c0e73f2197-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-da151d-6cfcff59d7-rqzfd\" (UID: \"ebc64c0e-74bd-4130-bf28-65c0e73f2197\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-da151d-6cfcff59d7-rqzfd" Oct 30 18:15:53.674499 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.674468 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-da151d-697b678888-d8zsf"] Oct 30 18:15:53.676011 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.675972 4455 cpu_manager.go:414] "containerMap: removing container" podUID="ba0a949f-05c3-4e1e-85f2-814291a242ab" containerName="cleanup" Oct 30 18:15:53.678285 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.678265 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba0a949f-05c3-4e1e-85f2-814291a242ab" containerName="cleanup" Oct 30 18:15:53.679763 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.679745 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba0a949f-05c3-4e1e-85f2-814291a242ab" containerName="cleanup" Oct 30 18:15:53.705346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.699238 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n472l\" (UniqueName: \"kubernetes.io/projected/ebc64c0e-74bd-4130-bf28-65c0e73f2197-kube-api-access-n472l\") pod \"payload-pre-processing-e2e-api-b-da151d-6cfcff59d7-rqzfd\" (UID: \"ebc64c0e-74bd-4130-bf28-65c0e73f2197\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-da151d-6cfcff59d7-rqzfd" Oct 30 18:15:53.739458 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.736088 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-da151d-697b678888-d8zsf"] Oct 30 18:15:53.739458 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.736217 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-da151d-697b678888-d8zsf" Oct 30 18:15:53.856561 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.856518 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f62dc6ab-f96a-49ed-9f75-158f121781d7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-da151d-697b678888-d8zsf\" (UID: \"f62dc6ab-f96a-49ed-9f75-158f121781d7\") " pod="openshift-ingress/payload-processing-e2e-api-b-da151d-697b678888-d8zsf" Oct 30 18:15:53.856752 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.856712 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htzfd\" (UniqueName: \"kubernetes.io/projected/f62dc6ab-f96a-49ed-9f75-158f121781d7-kube-api-access-htzfd\") pod \"payload-processing-e2e-api-b-da151d-697b678888-d8zsf\" (UID: \"f62dc6ab-f96a-49ed-9f75-158f121781d7\") " pod="openshift-ingress/payload-processing-e2e-api-b-da151d-697b678888-d8zsf" Oct 30 18:15:53.958784 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.958699 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f62dc6ab-f96a-49ed-9f75-158f121781d7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-da151d-697b678888-d8zsf\" (UID: \"f62dc6ab-f96a-49ed-9f75-158f121781d7\") " pod="openshift-ingress/payload-processing-e2e-api-b-da151d-697b678888-d8zsf" Oct 30 18:15:53.958972 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.958814 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htzfd\" (UniqueName: \"kubernetes.io/projected/f62dc6ab-f96a-49ed-9f75-158f121781d7-kube-api-access-htzfd\") pod \"payload-processing-e2e-api-b-da151d-697b678888-d8zsf\" (UID: \"f62dc6ab-f96a-49ed-9f75-158f121781d7\") " pod="openshift-ingress/payload-processing-e2e-api-b-da151d-697b678888-d8zsf" Oct 30 18:15:53.959525 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.959498 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f62dc6ab-f96a-49ed-9f75-158f121781d7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-da151d-697b678888-d8zsf\" (UID: \"f62dc6ab-f96a-49ed-9f75-158f121781d7\") " pod="openshift-ingress/payload-processing-e2e-api-b-da151d-697b678888-d8zsf" Oct 30 18:15:53.989144 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.989105 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htzfd\" (UniqueName: \"kubernetes.io/projected/f62dc6ab-f96a-49ed-9f75-158f121781d7-kube-api-access-htzfd\") pod \"payload-processing-e2e-api-b-da151d-697b678888-d8zsf\" (UID: \"f62dc6ab-f96a-49ed-9f75-158f121781d7\") " pod="openshift-ingress/payload-processing-e2e-api-b-da151d-697b678888-d8zsf" Oct 30 18:15:53.997355 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:53.997312 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da151d-6cfcff59d7-rqzfd" Oct 30 18:15:54.059775 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:54.059751 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-da151d-697b678888-d8zsf" Oct 30 18:15:54.305094 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:15:54.305063 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc64c0e_74bd_4130_bf28_65c0e73f2197.slice/crio-68a0380a66e2dbf9c9024b183840a3318515a7da47cc5baafbdb4b07531491ef WatchSource:0}: Error finding container 68a0380a66e2dbf9c9024b183840a3318515a7da47cc5baafbdb4b07531491ef: Status 404 returned error can't find the container with id 68a0380a66e2dbf9c9024b183840a3318515a7da47cc5baafbdb4b07531491ef Oct 30 18:15:54.307788 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:54.307760 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-da151d-6cfcff59d7-rqzfd"] Oct 30 18:15:54.403580 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:15:54.403555 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf62dc6ab_f96a_49ed_9f75_158f121781d7.slice/crio-85e2b50dc55df768ff348eb6ce2e161d0c7e9c75b8f742fb786ba60dc90e00b4 WatchSource:0}: Error finding container 85e2b50dc55df768ff348eb6ce2e161d0c7e9c75b8f742fb786ba60dc90e00b4: Status 404 returned error can't find the container with id 85e2b50dc55df768ff348eb6ce2e161d0c7e9c75b8f742fb786ba60dc90e00b4 Oct 30 18:15:54.406205 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:54.406181 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-da151d-697b678888-d8zsf"] Oct 30 18:15:55.253008 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:55.252956 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da151d-6cfcff59d7-rqzfd" event={"ID":"ebc64c0e-74bd-4130-bf28-65c0e73f2197","Type":"ContainerStarted","Data":"b27a1305268b2e2c1635e77f8e5c74c987b890dc479600e2ccd0a0472e8906f7"} Oct 30 18:15:55.253227 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:55.253024 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da151d-6cfcff59d7-rqzfd" event={"ID":"ebc64c0e-74bd-4130-bf28-65c0e73f2197","Type":"ContainerStarted","Data":"68a0380a66e2dbf9c9024b183840a3318515a7da47cc5baafbdb4b07531491ef"} Oct 30 18:15:55.255171 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:55.254076 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da151d-6cfcff59d7-rqzfd" Oct 30 18:15:55.257367 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:55.256410 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-da151d-697b678888-d8zsf" event={"ID":"f62dc6ab-f96a-49ed-9f75-158f121781d7","Type":"ContainerStarted","Data":"f8140bc47e28407013a8f2ca46ee3dff968a1e14218563cc74f095d5bfc4ac04"} Oct 30 18:15:55.257367 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:55.256458 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-da151d-697b678888-d8zsf" event={"ID":"f62dc6ab-f96a-49ed-9f75-158f121781d7","Type":"ContainerStarted","Data":"85e2b50dc55df768ff348eb6ce2e161d0c7e9c75b8f742fb786ba60dc90e00b4"} Oct 30 18:15:55.257367 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:55.257284 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-da151d-697b678888-d8zsf" Oct 30 18:15:55.303184 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:55.303118 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da151d-6cfcff59d7-rqzfd" podStartSLOduration=2.303089439 podStartE2EDuration="2.303089439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:55.298797715 +0000 UTC m=+1407.786434744" watchObservedRunningTime="2026-10-30 18:15:55.303089439 +0000 UTC m=+1407.790726473" Oct 30 18:15:55.344688 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:55.344623 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-da151d-697b678888-d8zsf" podStartSLOduration=2.34460385 podStartE2EDuration="2.34460385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:55.342636526 +0000 UTC m=+1407.830273560" watchObservedRunningTime="2026-10-30 18:15:55.34460385 +0000 UTC m=+1407.832240897" Oct 30 18:15:57.237632 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:57.234932 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw"] Oct 30 18:15:57.237632 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:15:57.235232 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" podUID="1c23fcb1-fecd-48a3-b3da-7dbe9873da88" containerName="istio-proxy" containerID="cri-o://4476b58ed44f9c158317daadf6f9f1e9788d8ce768ad0fc90a789a4cc2fe4158" gracePeriod=30 Oct 30 18:16:00.004348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:00.003417 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:00.608812 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:16:00.608767 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:16:01.014482 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:01.014416 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:01.108270 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:01.085508 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:16:02.034354 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:02.033847 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:16:02.782376 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:02.782057 4455 patch_prober.go:29] interesting pod/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 18:16:02.782376 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:02.782153 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" podUID="1c23fcb1-fecd-48a3-b3da-7dbe9873da88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 18:16:03.164112 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:03.164019 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:16:03.248750 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:03.248710 4455 generic.go:356] "Generic (PLEG): container finished" podID="1c23fcb1-fecd-48a3-b3da-7dbe9873da88" containerID="4476b58ed44f9c158317daadf6f9f1e9788d8ce768ad0fc90a789a4cc2fe4158" exitCode=0 Oct 30 18:16:03.249088 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:03.248986 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" event={"ID":"1c23fcb1-fecd-48a3-b3da-7dbe9873da88","Type":"ContainerDied","Data":"4476b58ed44f9c158317daadf6f9f1e9788d8ce768ad0fc90a789a4cc2fe4158"} Oct 30 18:16:03.510359 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:03.508878 4455 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:16:03.510359 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:03.508942 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" podUID="e972725e-489d-4f2a-a734-27d7eb570858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:03.997526 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:03.997060 4455 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:16:03.997526 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:03.997144 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" podUID="ac80ccea-a611-4de1-951c-479ad3d827a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:04.188358 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:04.183525 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:04.214353 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:04.213885 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:16:04.388754 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:04.388665 4455 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:16:04.389010 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:04.388979 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" podUID="70d7e6c8-8d0f-40f0-a533-58158542fb7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:04.765501 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:04.765467 4455 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:16:04.765769 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:04.765740 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" podUID="d7de539f-17e9-453c-af6b-e83fc7c91239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:04.937347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:04.936708 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" Oct 30 18:16:05.104402 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:05.028554 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:16:05.104402 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:05.051733 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-istio-envoy\") pod \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " Oct 30 18:16:05.104402 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:05.051803 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-istiod-ca-cert\") pod \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " Oct 30 18:16:05.104402 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:05.056573 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-istiod-ca-cert" pod "1c23fcb1-fecd-48a3-b3da-7dbe9873da88" (UID: "1c23fcb1-fecd-48a3-b3da-7dbe9873da88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:16:05.184086 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:05.173462 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-workload-socket\") pod \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " Oct 30 18:16:05.184086 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:05.173533 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-workload-certs\") pod \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " Oct 30 18:16:05.188083 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:05.187435 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-workload-certs" pod "1c23fcb1-fecd-48a3-b3da-7dbe9873da88" (UID: "1c23fcb1-fecd-48a3-b3da-7dbe9873da88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:16:05.206371 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:05.197437 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-istio-podinfo\") pod \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " Oct 30 18:16:05.206371 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:05.197577 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-credential-socket\") pod \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " Oct 30 18:16:05.206371 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:05.197641 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-kube-api-access-chn62\" (UniqueName: \"kubernetes.io/projected/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-kube-api-access-chn62\") pod \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " Oct 30 18:16:05.206371 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:05.197701 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-istio-data\") pod \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " Oct 30 18:16:05.206371 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:05.197764 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-istio-token\" (UniqueName: \"kubernetes.io/projected/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-istio-token\") pod \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\" (UID: \"1c23fcb1-fecd-48a3-b3da-7dbe9873da88\") " Oct 30 18:16:05.206371 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:05.198649 4455 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-istiod-ca-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:16:05.206371 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:05.198674 4455 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-workload-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:16:05.207740 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:05.207356 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-credential-socket" pod "1c23fcb1-fecd-48a3-b3da-7dbe9873da88" (UID: "1c23fcb1-fecd-48a3-b3da-7dbe9873da88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:16:05.231626 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:05.231576 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-istio-data" pod "1c23fcb1-fecd-48a3-b3da-7dbe9873da88" (UID: "1c23fcb1-fecd-48a3-b3da-7dbe9873da88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:16:05.295280 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:05.295206 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-istio-envoy" pod "1c23fcb1-fecd-48a3-b3da-7dbe9873da88" (UID: "1c23fcb1-fecd-48a3-b3da-7dbe9873da88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:16:05.313380 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:05.303717 4455 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-credential-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:16:05.313380 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:05.303757 4455 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-istio-data\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:16:05.313380 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:05.303771 4455 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-istio-envoy\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:16:05.319363 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:05.317722 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-workload-socket" pod "1c23fcb1-fecd-48a3-b3da-7dbe9873da88" (UID: "1c23fcb1-fecd-48a3-b3da-7dbe9873da88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:16:05.486111 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:05.466767 4455 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-workload-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:16:05.710414 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:05.680379 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" Oct 30 18:16:05.711348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:05.681457 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw" event={"ID":"1c23fcb1-fecd-48a3-b3da-7dbe9873da88","Type":"ContainerDied","Data":"3c57fac3f6782b2244baad0e4c33cdfec821486495a7499d045c19f62bc60afe"} Oct 30 18:16:05.761114 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:05.715138 4455 scope.go:122] "RemoveContainer" containerID="4476b58ed44f9c158317daadf6f9f1e9788d8ce768ad0fc90a789a4cc2fe4158" Oct 30 18:16:05.890340 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:05.890282 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-istio-token" pod "1c23fcb1-fecd-48a3-b3da-7dbe9873da88" (UID: "1c23fcb1-fecd-48a3-b3da-7dbe9873da88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:16:05.893943 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:05.893915 4455 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-istio-token\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:16:05.979396 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:05.977882 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-kube-api-access-chn62" pod "1c23fcb1-fecd-48a3-b3da-7dbe9873da88" (UID: "1c23fcb1-fecd-48a3-b3da-7dbe9873da88"). InnerVolumeSpecName "kube-api-access-chn62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:16:05.979396 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:05.977976 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-istio-podinfo" pod "1c23fcb1-fecd-48a3-b3da-7dbe9873da88" (UID: "1c23fcb1-fecd-48a3-b3da-7dbe9873da88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:16:06.104391 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:06.104356 4455 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-istio-podinfo\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:16:06.104633 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:06.104614 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chn62\" (UniqueName: \"kubernetes.io/projected/1c23fcb1-fecd-48a3-b3da-7dbe9873da88-kube-api-access-chn62\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:16:06.329054 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:06.328972 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw"] Oct 30 18:16:06.391431 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:06.356870 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-92cmw"] Oct 30 18:16:07.463991 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:07.463959 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da151d-6cfcff59d7-rqzfd" Oct 30 18:16:07.467529 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:07.467506 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-da151d-697b678888-d8zsf" Oct 30 18:16:07.651467 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:07.651431 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c23fcb1-fecd-48a3-b3da-7dbe9873da88" path="/var/lib/kubelet/pods/1c23fcb1-fecd-48a3-b3da-7dbe9873da88/volumes" Oct 30 18:16:23.101918 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.101886 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf"] Oct 30 18:16:23.102672 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.102658 4455 cpu_manager.go:414] "containerMap: removing container" podUID="1c23fcb1-fecd-48a3-b3da-7dbe9873da88" containerName="istio-proxy" Oct 30 18:16:23.102734 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.102673 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c23fcb1-fecd-48a3-b3da-7dbe9873da88" containerName="istio-proxy" Oct 30 18:16:23.102801 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.102791 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c23fcb1-fecd-48a3-b3da-7dbe9873da88" containerName="istio-proxy" Oct 30 18:16:23.107621 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.107599 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" Oct 30 18:16:23.126072 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.126039 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c6b2120f-openshift-default-dockercfg-xt8vs\"" Oct 30 18:16:23.145181 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.145150 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf"] Oct 30 18:16:23.197402 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.197369 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87894ec9-1772-4a1d-9b7d-e6489df743a4-workload-socket\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" Oct 30 18:16:23.197588 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.197414 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pg5d\" (UniqueName: \"kubernetes.io/projected/87894ec9-1772-4a1d-9b7d-e6489df743a4-kube-api-access-6pg5d\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" Oct 30 18:16:23.197664 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.197591 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87894ec9-1772-4a1d-9b7d-e6489df743a4-istio-data\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" Oct 30 18:16:23.197664 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.197645 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87894ec9-1772-4a1d-9b7d-e6489df743a4-credential-socket\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" Oct 30 18:16:23.197775 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.197703 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87894ec9-1772-4a1d-9b7d-e6489df743a4-istio-token\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" Oct 30 18:16:23.197816 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.197772 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87894ec9-1772-4a1d-9b7d-e6489df743a4-istio-podinfo\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" Oct 30 18:16:23.197862 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.197821 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87894ec9-1772-4a1d-9b7d-e6489df743a4-istio-envoy\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" Oct 30 18:16:23.197940 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.197925 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87894ec9-1772-4a1d-9b7d-e6489df743a4-istiod-ca-cert\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" Oct 30 18:16:23.197997 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.197958 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87894ec9-1772-4a1d-9b7d-e6489df743a4-workload-certs\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" Oct 30 18:16:23.299584 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.299552 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pg5d\" (UniqueName: \"kubernetes.io/projected/87894ec9-1772-4a1d-9b7d-e6489df743a4-kube-api-access-6pg5d\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" Oct 30 18:16:23.299778 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.299655 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87894ec9-1772-4a1d-9b7d-e6489df743a4-istio-data\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" Oct 30 18:16:23.299778 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.299696 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87894ec9-1772-4a1d-9b7d-e6489df743a4-credential-socket\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" Oct 30 18:16:23.299778 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.299741 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87894ec9-1772-4a1d-9b7d-e6489df743a4-istio-token\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" Oct 30 18:16:23.299778 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.299775 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87894ec9-1772-4a1d-9b7d-e6489df743a4-istio-podinfo\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" Oct 30 18:16:23.300007 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.299847 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87894ec9-1772-4a1d-9b7d-e6489df743a4-istio-envoy\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" Oct 30 18:16:23.300007 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.299904 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87894ec9-1772-4a1d-9b7d-e6489df743a4-istiod-ca-cert\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" Oct 30 18:16:23.300007 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.300000 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87894ec9-1772-4a1d-9b7d-e6489df743a4-workload-certs\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" Oct 30 18:16:23.300175 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.300132 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87894ec9-1772-4a1d-9b7d-e6489df743a4-istio-data\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" Oct 30 18:16:23.300296 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.300233 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87894ec9-1772-4a1d-9b7d-e6489df743a4-workload-socket\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" Oct 30 18:16:23.300643 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.300615 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87894ec9-1772-4a1d-9b7d-e6489df743a4-credential-socket\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" Oct 30 18:16:23.300763 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.300652 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87894ec9-1772-4a1d-9b7d-e6489df743a4-workload-certs\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" Oct 30 18:16:23.300763 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.300757 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87894ec9-1772-4a1d-9b7d-e6489df743a4-workload-socket\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" Oct 30 18:16:23.300908 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.300882 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87894ec9-1772-4a1d-9b7d-e6489df743a4-istiod-ca-cert\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" Oct 30 18:16:23.302950 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.302919 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87894ec9-1772-4a1d-9b7d-e6489df743a4-istio-envoy\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" Oct 30 18:16:23.303050 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.303033 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87894ec9-1772-4a1d-9b7d-e6489df743a4-istio-podinfo\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" Oct 30 18:16:23.324932 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.324900 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pg5d\" (UniqueName: \"kubernetes.io/projected/87894ec9-1772-4a1d-9b7d-e6489df743a4-kube-api-access-6pg5d\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" Oct 30 18:16:23.325088 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.325068 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87894ec9-1772-4a1d-9b7d-e6489df743a4-istio-token\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" Oct 30 18:16:23.422984 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.422897 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" Oct 30 18:16:23.746676 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:16:23.746645 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87894ec9_1772_4a1d_9b7d_e6489df743a4.slice/crio-0387748736b163530244a52dc4abfcdeb98a5a0676dd29918fe2ae1ba0d8067e WatchSource:0}: Error finding container 0387748736b163530244a52dc4abfcdeb98a5a0676dd29918fe2ae1ba0d8067e: Status 404 returned error can't find the container with id 0387748736b163530244a52dc4abfcdeb98a5a0676dd29918fe2ae1ba0d8067e Oct 30 18:16:23.748792 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.748765 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf"] Oct 30 18:16:23.749586 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.749555 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:16:23.749658 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.749620 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:16:23.749658 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:23.749652 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:16:24.238921 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:24.238881 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" event={"ID":"87894ec9-1772-4a1d-9b7d-e6489df743a4","Type":"ContainerStarted","Data":"c8384c9571fc8961ee5feb5c0f55c39442abf65ca37b0b306e5a5e3e33490a5f"} Oct 30 18:16:24.239380 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:24.238949 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" event={"ID":"87894ec9-1772-4a1d-9b7d-e6489df743a4","Type":"ContainerStarted","Data":"0387748736b163530244a52dc4abfcdeb98a5a0676dd29918fe2ae1ba0d8067e"} Oct 30 18:16:24.295347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:24.292079 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" podStartSLOduration=1.2920586059999999 podStartE2EDuration="1.292058606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:24.291755188 +0000 UTC m=+1436.779392235" watchObservedRunningTime="2026-10-30 18:16:24.292058606 +0000 UTC m=+1436.779695641" Oct 30 18:16:24.423829 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:24.423784 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" Oct 30 18:16:24.425336 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:24.425298 4455 patch_prober.go:29] interesting pod/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:16:24.425447 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:24.425372 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" podUID="87894ec9-1772-4a1d-9b7d-e6489df743a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 18:16:25.423639 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:25.423601 4455 patch_prober.go:29] interesting pod/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:16:25.424048 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:25.423663 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" podUID="87894ec9-1772-4a1d-9b7d-e6489df743a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 18:16:26.423718 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:26.423685 4455 patch_prober.go:29] interesting pod/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:16:26.424136 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:26.423754 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" podUID="87894ec9-1772-4a1d-9b7d-e6489df743a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 18:16:27.423787 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:27.423750 4455 patch_prober.go:29] interesting pod/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:16:27.424228 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:27.423811 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" podUID="87894ec9-1772-4a1d-9b7d-e6489df743a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 18:16:28.423709 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:28.423677 4455 patch_prober.go:29] interesting pod/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:16:28.423938 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:28.423915 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" podUID="87894ec9-1772-4a1d-9b7d-e6489df743a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 18:16:29.423720 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:29.423636 4455 patch_prober.go:29] interesting pod/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:16:29.423939 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:29.423917 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" podUID="87894ec9-1772-4a1d-9b7d-e6489df743a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 18:16:30.391620 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:16:30.391564 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:16:30.619260 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:30.619221 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" Oct 30 18:16:30.619726 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:30.619707 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" Oct 30 18:16:30.620672 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:16:30.620644 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" Oct 30 18:17:00.256670 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:00.256632 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-r6l4c"] Oct 30 18:17:00.293518 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:00.293475 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-l4frw"] Oct 30 18:17:00.419603 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:17:00.419557 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:17:04.762198 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:04.762160 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c8f010-6c8445bbdb-fxgzp"] Oct 30 18:17:05.257698 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:05.257641 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-r6l4c" podUID="dcef406f-9a07-4803-b883-aac4b334de0c" containerName="payload-processing" containerID="cri-o://470c7e5dec6fb249ce2a899131286afb4aa46ceeba34fca23bfa4c6cbfce8eb5" gracePeriod=25 Oct 30 18:17:05.294511 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:05.294465 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-l4frw" podUID="0bd62dee-70b4-4f07-9f41-3186f5550228" containerName="payload-pre-processing" containerID="cri-o://cdf57d5e26ad49d89da6fbc94fadefb5199024081ac5a9943365e222c42636f2" gracePeriod=25 Oct 30 18:17:05.885380 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:05.885312 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-l4frw" podUID="0bd62dee-70b4-4f07-9f41-3186f5550228" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:17:05.886078 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:05.885837 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-r6l4c" podUID="dcef406f-9a07-4803-b883-aac4b334de0c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:17:09.764543 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:09.764306 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c8f010-6c8445bbdb-fxgzp" podUID="14a96f10-41c5-44ba-b8ca-86d9fb9520cb" containerName="payload-processing" containerID="cri-o://60e7740bfe7e0e9997b297611c30cd13b3a131ee5c20319a85933eac680fd8c4" gracePeriod=25 Oct 30 18:17:10.148065 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:10.148040 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c8f010-6c8445bbdb-fxgzp" Oct 30 18:17:10.223406 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:10.222770 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14a96f10-41c5-44ba-b8ca-86d9fb9520cb-kube-api-access-dqbd5\" (UniqueName: \"kubernetes.io/projected/14a96f10-41c5-44ba-b8ca-86d9fb9520cb-kube-api-access-dqbd5\") pod \"14a96f10-41c5-44ba-b8ca-86d9fb9520cb\" (UID: \"14a96f10-41c5-44ba-b8ca-86d9fb9520cb\") " Oct 30 18:17:10.223406 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:10.222865 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14a96f10-41c5-44ba-b8ca-86d9fb9520cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14a96f10-41c5-44ba-b8ca-86d9fb9520cb-plugins-config-volume\") pod \"14a96f10-41c5-44ba-b8ca-86d9fb9520cb\" (UID: \"14a96f10-41c5-44ba-b8ca-86d9fb9520cb\") " Oct 30 18:17:10.223824 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:10.223800 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a96f10-41c5-44ba-b8ca-86d9fb9520cb-plugins-config-volume" pod "14a96f10-41c5-44ba-b8ca-86d9fb9520cb" (UID: "14a96f10-41c5-44ba-b8ca-86d9fb9520cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:17:10.226775 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:10.226742 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a96f10-41c5-44ba-b8ca-86d9fb9520cb-kube-api-access-dqbd5" pod "14a96f10-41c5-44ba-b8ca-86d9fb9520cb" (UID: "14a96f10-41c5-44ba-b8ca-86d9fb9520cb"). InnerVolumeSpecName "kube-api-access-dqbd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:10.325079 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:10.325034 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqbd5\" (UniqueName: \"kubernetes.io/projected/14a96f10-41c5-44ba-b8ca-86d9fb9520cb-kube-api-access-dqbd5\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:17:10.325079 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:10.325076 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14a96f10-41c5-44ba-b8ca-86d9fb9520cb-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:17:10.700946 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:10.700906 4455 generic.go:356] "Generic (PLEG): container finished" podID="14a96f10-41c5-44ba-b8ca-86d9fb9520cb" containerID="60e7740bfe7e0e9997b297611c30cd13b3a131ee5c20319a85933eac680fd8c4" exitCode=0 Oct 30 18:17:10.700946 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:10.700952 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c8f010-6c8445bbdb-fxgzp" Oct 30 18:17:10.701234 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:10.700999 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c8f010-6c8445bbdb-fxgzp" event={"ID":"14a96f10-41c5-44ba-b8ca-86d9fb9520cb","Type":"ContainerDied","Data":"60e7740bfe7e0e9997b297611c30cd13b3a131ee5c20319a85933eac680fd8c4"} Oct 30 18:17:10.701234 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:10.701063 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c8f010-6c8445bbdb-fxgzp" event={"ID":"14a96f10-41c5-44ba-b8ca-86d9fb9520cb","Type":"ContainerDied","Data":"8ecb42d9f6b17cf56050df15e83353cd3ed8ced8dbd38f9c2e30ace4c6b55211"} Oct 30 18:17:10.701234 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:10.701092 4455 scope.go:122] "RemoveContainer" containerID="60e7740bfe7e0e9997b297611c30cd13b3a131ee5c20319a85933eac680fd8c4" Oct 30 18:17:10.716823 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:10.716800 4455 scope.go:122] "RemoveContainer" containerID="60e7740bfe7e0e9997b297611c30cd13b3a131ee5c20319a85933eac680fd8c4" Oct 30 18:17:10.717164 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:17:10.717137 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60e7740bfe7e0e9997b297611c30cd13b3a131ee5c20319a85933eac680fd8c4\": container with ID starting with 60e7740bfe7e0e9997b297611c30cd13b3a131ee5c20319a85933eac680fd8c4 not found: ID does not exist" containerID="60e7740bfe7e0e9997b297611c30cd13b3a131ee5c20319a85933eac680fd8c4" Oct 30 18:17:10.717273 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:10.717174 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60e7740bfe7e0e9997b297611c30cd13b3a131ee5c20319a85933eac680fd8c4"} err="failed to get container status \"60e7740bfe7e0e9997b297611c30cd13b3a131ee5c20319a85933eac680fd8c4\": rpc error: code = NotFound desc = could not find container \"60e7740bfe7e0e9997b297611c30cd13b3a131ee5c20319a85933eac680fd8c4\": container with ID starting with 60e7740bfe7e0e9997b297611c30cd13b3a131ee5c20319a85933eac680fd8c4 not found: ID does not exist" Oct 30 18:17:10.772738 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:10.772701 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c8f010-6c8445bbdb-fxgzp"] Oct 30 18:17:10.785718 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:10.785682 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c8f010-6c8445bbdb-fxgzp"] Oct 30 18:17:11.663383 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:11.659741 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14a96f10-41c5-44ba-b8ca-86d9fb9520cb" path="/var/lib/kubelet/pods/14a96f10-41c5-44ba-b8ca-86d9fb9520cb/volumes" Oct 30 18:17:15.076350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:15.064255 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:17:16.094352 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:16.086729 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-l4frw" podUID="0bd62dee-70b4-4f07-9f41-3186f5550228" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:17:16.099407 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:16.086675 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:17:16.125352 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:16.119475 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:17:16.125661 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:16.125631 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-r6l4c" podUID="dcef406f-9a07-4803-b883-aac4b334de0c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:17:17.100533 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:17.100470 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:17.346132 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:17.346084 4455 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:18.127352 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:18.119786 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:19.122348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:19.121437 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:17:19.198669 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:19.198636 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:17:25.628529 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:25.628502 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-r6l4c" Oct 30 18:17:25.634426 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:25.634404 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-l4frw" Oct 30 18:17:25.722755 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:25.722723 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcef406f-9a07-4803-b883-aac4b334de0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcef406f-9a07-4803-b883-aac4b334de0c-plugins-config-volume\") pod \"dcef406f-9a07-4803-b883-aac4b334de0c\" (UID: \"dcef406f-9a07-4803-b883-aac4b334de0c\") " Oct 30 18:17:25.722950 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:25.722919 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcef406f-9a07-4803-b883-aac4b334de0c-kube-api-access-jr54b\" (UniqueName: \"kubernetes.io/projected/dcef406f-9a07-4803-b883-aac4b334de0c-kube-api-access-jr54b\") pod \"dcef406f-9a07-4803-b883-aac4b334de0c\" (UID: \"dcef406f-9a07-4803-b883-aac4b334de0c\") " Oct 30 18:17:25.723017 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:25.722966 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bd62dee-70b4-4f07-9f41-3186f5550228-kube-api-access-txgqn\" (UniqueName: \"kubernetes.io/projected/0bd62dee-70b4-4f07-9f41-3186f5550228-kube-api-access-txgqn\") pod \"0bd62dee-70b4-4f07-9f41-3186f5550228\" (UID: \"0bd62dee-70b4-4f07-9f41-3186f5550228\") " Oct 30 18:17:25.723071 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:25.723017 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bd62dee-70b4-4f07-9f41-3186f5550228-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd62dee-70b4-4f07-9f41-3186f5550228-plugins-config-volume\") pod \"0bd62dee-70b4-4f07-9f41-3186f5550228\" (UID: \"0bd62dee-70b4-4f07-9f41-3186f5550228\") " Oct 30 18:17:25.725211 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:25.725181 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bd62dee-70b4-4f07-9f41-3186f5550228-plugins-config-volume" pod "0bd62dee-70b4-4f07-9f41-3186f5550228" (UID: "0bd62dee-70b4-4f07-9f41-3186f5550228"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:17:25.725943 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:25.725794 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcef406f-9a07-4803-b883-aac4b334de0c-plugins-config-volume" pod "dcef406f-9a07-4803-b883-aac4b334de0c" (UID: "dcef406f-9a07-4803-b883-aac4b334de0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:17:25.726083 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:25.726021 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcef406f-9a07-4803-b883-aac4b334de0c-kube-api-access-jr54b" pod "dcef406f-9a07-4803-b883-aac4b334de0c" (UID: "dcef406f-9a07-4803-b883-aac4b334de0c"). InnerVolumeSpecName "kube-api-access-jr54b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:25.727927 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:25.727898 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bd62dee-70b4-4f07-9f41-3186f5550228-kube-api-access-txgqn" pod "0bd62dee-70b4-4f07-9f41-3186f5550228" (UID: "0bd62dee-70b4-4f07-9f41-3186f5550228"). InnerVolumeSpecName "kube-api-access-txgqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:25.825510 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:25.825474 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr54b\" (UniqueName: \"kubernetes.io/projected/dcef406f-9a07-4803-b883-aac4b334de0c-kube-api-access-jr54b\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:17:25.825510 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:25.825503 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txgqn\" (UniqueName: \"kubernetes.io/projected/0bd62dee-70b4-4f07-9f41-3186f5550228-kube-api-access-txgqn\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:17:25.825510 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:25.825513 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd62dee-70b4-4f07-9f41-3186f5550228-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:17:25.825740 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:25.825522 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcef406f-9a07-4803-b883-aac4b334de0c-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:17:25.886980 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:25.886897 4455 generic.go:356] "Generic (PLEG): container finished" podID="0bd62dee-70b4-4f07-9f41-3186f5550228" containerID="cdf57d5e26ad49d89da6fbc94fadefb5199024081ac5a9943365e222c42636f2" exitCode=0 Oct 30 18:17:25.886980 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:25.886944 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-l4frw" Oct 30 18:17:25.887234 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:25.886984 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-l4frw" event={"ID":"0bd62dee-70b4-4f07-9f41-3186f5550228","Type":"ContainerDied","Data":"cdf57d5e26ad49d89da6fbc94fadefb5199024081ac5a9943365e222c42636f2"} Oct 30 18:17:25.887234 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:25.887047 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-l4frw" event={"ID":"0bd62dee-70b4-4f07-9f41-3186f5550228","Type":"ContainerDied","Data":"2e5121796164a7a15bfd78d3fc33c0578559b9cf9212f35f22852de46857d1fa"} Oct 30 18:17:25.887234 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:25.887079 4455 scope.go:122] "RemoveContainer" containerID="cdf57d5e26ad49d89da6fbc94fadefb5199024081ac5a9943365e222c42636f2" Oct 30 18:17:25.888881 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:25.888854 4455 generic.go:356] "Generic (PLEG): container finished" podID="dcef406f-9a07-4803-b883-aac4b334de0c" containerID="470c7e5dec6fb249ce2a899131286afb4aa46ceeba34fca23bfa4c6cbfce8eb5" exitCode=0 Oct 30 18:17:25.889018 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:25.888946 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-r6l4c" event={"ID":"dcef406f-9a07-4803-b883-aac4b334de0c","Type":"ContainerDied","Data":"470c7e5dec6fb249ce2a899131286afb4aa46ceeba34fca23bfa4c6cbfce8eb5"} Oct 30 18:17:25.889018 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:25.889002 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-r6l4c" event={"ID":"dcef406f-9a07-4803-b883-aac4b334de0c","Type":"ContainerDied","Data":"a33eb1d5c4d2889ce490c5211203c1bb4f53fa34c3e3025cfdf5852863735d65"} Oct 30 18:17:25.889136 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:25.889096 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-r6l4c" Oct 30 18:17:25.899576 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:25.899557 4455 scope.go:122] "RemoveContainer" containerID="cdf57d5e26ad49d89da6fbc94fadefb5199024081ac5a9943365e222c42636f2" Oct 30 18:17:25.899878 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:17:25.899855 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdf57d5e26ad49d89da6fbc94fadefb5199024081ac5a9943365e222c42636f2\": container with ID starting with cdf57d5e26ad49d89da6fbc94fadefb5199024081ac5a9943365e222c42636f2 not found: ID does not exist" containerID="cdf57d5e26ad49d89da6fbc94fadefb5199024081ac5a9943365e222c42636f2" Oct 30 18:17:25.899982 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:25.899889 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdf57d5e26ad49d89da6fbc94fadefb5199024081ac5a9943365e222c42636f2"} err="failed to get container status \"cdf57d5e26ad49d89da6fbc94fadefb5199024081ac5a9943365e222c42636f2\": rpc error: code = NotFound desc = could not find container \"cdf57d5e26ad49d89da6fbc94fadefb5199024081ac5a9943365e222c42636f2\": container with ID starting with cdf57d5e26ad49d89da6fbc94fadefb5199024081ac5a9943365e222c42636f2 not found: ID does not exist" Oct 30 18:17:25.899982 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:25.899914 4455 scope.go:122] "RemoveContainer" containerID="470c7e5dec6fb249ce2a899131286afb4aa46ceeba34fca23bfa4c6cbfce8eb5" Oct 30 18:17:25.910956 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:25.910932 4455 scope.go:122] "RemoveContainer" containerID="470c7e5dec6fb249ce2a899131286afb4aa46ceeba34fca23bfa4c6cbfce8eb5" Oct 30 18:17:25.911279 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:17:25.911259 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"470c7e5dec6fb249ce2a899131286afb4aa46ceeba34fca23bfa4c6cbfce8eb5\": container with ID starting with 470c7e5dec6fb249ce2a899131286afb4aa46ceeba34fca23bfa4c6cbfce8eb5 not found: ID does not exist" containerID="470c7e5dec6fb249ce2a899131286afb4aa46ceeba34fca23bfa4c6cbfce8eb5" Oct 30 18:17:25.911384 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:25.911289 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"470c7e5dec6fb249ce2a899131286afb4aa46ceeba34fca23bfa4c6cbfce8eb5"} err="failed to get container status \"470c7e5dec6fb249ce2a899131286afb4aa46ceeba34fca23bfa4c6cbfce8eb5\": rpc error: code = NotFound desc = could not find container \"470c7e5dec6fb249ce2a899131286afb4aa46ceeba34fca23bfa4c6cbfce8eb5\": container with ID starting with 470c7e5dec6fb249ce2a899131286afb4aa46ceeba34fca23bfa4c6cbfce8eb5 not found: ID does not exist" Oct 30 18:17:25.945828 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:25.945795 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-l4frw"] Oct 30 18:17:25.949805 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:25.949778 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-l4frw"] Oct 30 18:17:25.986261 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:25.986229 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-r6l4c"] Oct 30 18:17:25.991801 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:25.991775 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-r6l4c"] Oct 30 18:17:27.636481 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:27.636453 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bd62dee-70b4-4f07-9f41-3186f5550228" path="/var/lib/kubelet/pods/0bd62dee-70b4-4f07-9f41-3186f5550228/volumes" Oct 30 18:17:27.636998 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:27.636983 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcef406f-9a07-4803-b883-aac4b334de0c" path="/var/lib/kubelet/pods/dcef406f-9a07-4803-b883-aac4b334de0c/volumes" Oct 30 18:17:27.662940 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:27.662910 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq"] Oct 30 18:17:27.663188 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:27.663162 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" podUID="9bf6d8f4-3ec1-449f-9399-e2e238d5ee11" containerName="maas-api" containerID="cri-o://31381ecb3ef748cfd2995d4f1b6bf1bc25b9af849114a2015f3a7e09282afafe" gracePeriod=30 Oct 30 18:17:28.048347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:28.047547 4455 generic.go:356] "Generic (PLEG): container finished" podID="9bf6d8f4-3ec1-449f-9399-e2e238d5ee11" containerID="31381ecb3ef748cfd2995d4f1b6bf1bc25b9af849114a2015f3a7e09282afafe" exitCode=0 Oct 30 18:17:28.048347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:28.047622 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" event={"ID":"9bf6d8f4-3ec1-449f-9399-e2e238d5ee11","Type":"ContainerDied","Data":"31381ecb3ef748cfd2995d4f1b6bf1bc25b9af849114a2015f3a7e09282afafe"} Oct 30 18:17:28.048347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:28.047678 4455 scope.go:122] "RemoveContainer" containerID="199472a03476a3106ef72b622a543763804d52f8cdd439a718fb52cb6810c28a" Oct 30 18:17:28.179347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:28.178733 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp"] Oct 30 18:17:28.190353 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:28.189616 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp" podUID="52e2e782-aab7-463d-b91a-e37a774091ab" containerName="maas-api" containerID="cri-o://d114d95ea33ea3e2f4eff1939e0452977bd1ddeed14b00411742c7a8478297b7" gracePeriod=30 Oct 30 18:17:28.690378 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:28.690005 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" Oct 30 18:17:28.773989 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:28.772449 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bf6d8f4-3ec1-449f-9399-e2e238d5ee11-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bf6d8f4-3ec1-449f-9399-e2e238d5ee11-maas-api-tls\") pod \"9bf6d8f4-3ec1-449f-9399-e2e238d5ee11\" (UID: \"9bf6d8f4-3ec1-449f-9399-e2e238d5ee11\") " Oct 30 18:17:28.773989 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:28.772631 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bf6d8f4-3ec1-449f-9399-e2e238d5ee11-kube-api-access-nwv24\" (UniqueName: \"kubernetes.io/projected/9bf6d8f4-3ec1-449f-9399-e2e238d5ee11-kube-api-access-nwv24\") pod \"9bf6d8f4-3ec1-449f-9399-e2e238d5ee11\" (UID: \"9bf6d8f4-3ec1-449f-9399-e2e238d5ee11\") " Oct 30 18:17:28.773989 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:28.772670 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bf6d8f4-3ec1-449f-9399-e2e238d5ee11-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bf6d8f4-3ec1-449f-9399-e2e238d5ee11-openshift-service-ca\") pod \"9bf6d8f4-3ec1-449f-9399-e2e238d5ee11\" (UID: \"9bf6d8f4-3ec1-449f-9399-e2e238d5ee11\") " Oct 30 18:17:28.773989 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:28.772815 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bf6d8f4-3ec1-449f-9399-e2e238d5ee11-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bf6d8f4-3ec1-449f-9399-e2e238d5ee11-metrics-certs\") pod \"9bf6d8f4-3ec1-449f-9399-e2e238d5ee11\" (UID: \"9bf6d8f4-3ec1-449f-9399-e2e238d5ee11\") " Oct 30 18:17:28.775888 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:28.775858 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bf6d8f4-3ec1-449f-9399-e2e238d5ee11-openshift-service-ca" pod "9bf6d8f4-3ec1-449f-9399-e2e238d5ee11" (UID: "9bf6d8f4-3ec1-449f-9399-e2e238d5ee11"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:17:28.832522 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:28.830981 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bf6d8f4-3ec1-449f-9399-e2e238d5ee11-metrics-certs" pod "9bf6d8f4-3ec1-449f-9399-e2e238d5ee11" (UID: "9bf6d8f4-3ec1-449f-9399-e2e238d5ee11"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:17:28.832522 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:28.831126 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bf6d8f4-3ec1-449f-9399-e2e238d5ee11-kube-api-access-nwv24" pod "9bf6d8f4-3ec1-449f-9399-e2e238d5ee11" (UID: "9bf6d8f4-3ec1-449f-9399-e2e238d5ee11"). InnerVolumeSpecName "kube-api-access-nwv24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:28.841615 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:28.841561 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bf6d8f4-3ec1-449f-9399-e2e238d5ee11-maas-api-tls" pod "9bf6d8f4-3ec1-449f-9399-e2e238d5ee11" (UID: "9bf6d8f4-3ec1-449f-9399-e2e238d5ee11"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:17:28.877232 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:28.877029 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwv24\" (UniqueName: \"kubernetes.io/projected/9bf6d8f4-3ec1-449f-9399-e2e238d5ee11-kube-api-access-nwv24\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:17:28.877232 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:28.877061 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bf6d8f4-3ec1-449f-9399-e2e238d5ee11-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:17:28.877232 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:28.877075 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bf6d8f4-3ec1-449f-9399-e2e238d5ee11-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:17:28.877232 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:28.877088 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bf6d8f4-3ec1-449f-9399-e2e238d5ee11-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:17:29.033119 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:29.033091 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp" Oct 30 18:17:29.181360 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:29.181205 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" event={"ID":"9bf6d8f4-3ec1-449f-9399-e2e238d5ee11","Type":"ContainerDied","Data":"62291a4d69baa46e410831d1e4e6fdd4b08340bdf99959b270137579bbc43e1b"} Oct 30 18:17:29.181360 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:29.181197 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq" Oct 30 18:17:29.181360 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:29.181306 4455 scope.go:122] "RemoveContainer" containerID="31381ecb3ef748cfd2995d4f1b6bf1bc25b9af849114a2015f3a7e09282afafe" Oct 30 18:17:29.184942 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:29.184857 4455 generic.go:356] "Generic (PLEG): container finished" podID="52e2e782-aab7-463d-b91a-e37a774091ab" containerID="d114d95ea33ea3e2f4eff1939e0452977bd1ddeed14b00411742c7a8478297b7" exitCode=0 Oct 30 18:17:29.185402 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:29.185109 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp" event={"ID":"52e2e782-aab7-463d-b91a-e37a774091ab","Type":"ContainerDied","Data":"d114d95ea33ea3e2f4eff1939e0452977bd1ddeed14b00411742c7a8478297b7"} Oct 30 18:17:29.185402 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:29.185167 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp" event={"ID":"52e2e782-aab7-463d-b91a-e37a774091ab","Type":"ContainerDied","Data":"58f2aecc69b8a8cf840969989f262437e137e1a056f32ce74058e8edd8c24535"} Oct 30 18:17:29.185402 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:29.185241 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp" Oct 30 18:17:29.195940 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:29.195911 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52e2e782-aab7-463d-b91a-e37a774091ab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52e2e782-aab7-463d-b91a-e37a774091ab-maas-api-tls\") pod \"52e2e782-aab7-463d-b91a-e37a774091ab\" (UID: \"52e2e782-aab7-463d-b91a-e37a774091ab\") " Oct 30 18:17:29.196047 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:29.196026 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52e2e782-aab7-463d-b91a-e37a774091ab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/52e2e782-aab7-463d-b91a-e37a774091ab-metrics-certs\") pod \"52e2e782-aab7-463d-b91a-e37a774091ab\" (UID: \"52e2e782-aab7-463d-b91a-e37a774091ab\") " Oct 30 18:17:29.196123 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:29.196111 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52e2e782-aab7-463d-b91a-e37a774091ab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52e2e782-aab7-463d-b91a-e37a774091ab-openshift-service-ca\") pod \"52e2e782-aab7-463d-b91a-e37a774091ab\" (UID: \"52e2e782-aab7-463d-b91a-e37a774091ab\") " Oct 30 18:17:29.196178 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:29.196165 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52e2e782-aab7-463d-b91a-e37a774091ab-kube-api-access-csk8q\" (UniqueName: \"kubernetes.io/projected/52e2e782-aab7-463d-b91a-e37a774091ab-kube-api-access-csk8q\") pod \"52e2e782-aab7-463d-b91a-e37a774091ab\" (UID: \"52e2e782-aab7-463d-b91a-e37a774091ab\") " Oct 30 18:17:29.196692 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:29.196662 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52e2e782-aab7-463d-b91a-e37a774091ab-openshift-service-ca" pod "52e2e782-aab7-463d-b91a-e37a774091ab" (UID: "52e2e782-aab7-463d-b91a-e37a774091ab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:17:29.196796 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:29.196765 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52e2e782-aab7-463d-b91a-e37a774091ab-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:17:29.208734 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:29.208470 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52e2e782-aab7-463d-b91a-e37a774091ab-kube-api-access-csk8q" pod "52e2e782-aab7-463d-b91a-e37a774091ab" (UID: "52e2e782-aab7-463d-b91a-e37a774091ab"). InnerVolumeSpecName "kube-api-access-csk8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:29.208734 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:29.208549 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52e2e782-aab7-463d-b91a-e37a774091ab-metrics-certs" pod "52e2e782-aab7-463d-b91a-e37a774091ab" (UID: "52e2e782-aab7-463d-b91a-e37a774091ab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:17:29.209963 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:29.209934 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52e2e782-aab7-463d-b91a-e37a774091ab-maas-api-tls" pod "52e2e782-aab7-463d-b91a-e37a774091ab" (UID: "52e2e782-aab7-463d-b91a-e37a774091ab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:17:29.235259 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:29.235205 4455 scope.go:122] "RemoveContainer" containerID="d114d95ea33ea3e2f4eff1939e0452977bd1ddeed14b00411742c7a8478297b7" Oct 30 18:17:29.245508 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:29.245438 4455 scope.go:122] "RemoveContainer" containerID="d114d95ea33ea3e2f4eff1939e0452977bd1ddeed14b00411742c7a8478297b7" Oct 30 18:17:29.245728 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:17:29.245710 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d114d95ea33ea3e2f4eff1939e0452977bd1ddeed14b00411742c7a8478297b7\": container with ID starting with d114d95ea33ea3e2f4eff1939e0452977bd1ddeed14b00411742c7a8478297b7 not found: ID does not exist" containerID="d114d95ea33ea3e2f4eff1939e0452977bd1ddeed14b00411742c7a8478297b7" Oct 30 18:17:29.245793 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:29.245737 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d114d95ea33ea3e2f4eff1939e0452977bd1ddeed14b00411742c7a8478297b7"} err="failed to get container status \"d114d95ea33ea3e2f4eff1939e0452977bd1ddeed14b00411742c7a8478297b7\": rpc error: code = NotFound desc = could not find container \"d114d95ea33ea3e2f4eff1939e0452977bd1ddeed14b00411742c7a8478297b7\": container with ID starting with d114d95ea33ea3e2f4eff1939e0452977bd1ddeed14b00411742c7a8478297b7 not found: ID does not exist" Oct 30 18:17:29.260586 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:29.260526 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq"] Oct 30 18:17:29.272563 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:29.272539 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-2cfpq"] Oct 30 18:17:29.298353 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:29.298286 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52e2e782-aab7-463d-b91a-e37a774091ab-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:17:29.298353 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:29.298344 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csk8q\" (UniqueName: \"kubernetes.io/projected/52e2e782-aab7-463d-b91a-e37a774091ab-kube-api-access-csk8q\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:17:29.298541 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:29.298363 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52e2e782-aab7-463d-b91a-e37a774091ab-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:17:29.540460 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:29.540372 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp"] Oct 30 18:17:29.544125 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:29.544103 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c8f010-d5c5dc5b6-k7dgp"] Oct 30 18:17:29.635291 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:29.635261 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52e2e782-aab7-463d-b91a-e37a774091ab" path="/var/lib/kubelet/pods/52e2e782-aab7-463d-b91a-e37a774091ab/volumes" Oct 30 18:17:29.635801 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:29.635786 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bf6d8f4-3ec1-449f-9399-e2e238d5ee11" path="/var/lib/kubelet/pods/9bf6d8f4-3ec1-449f-9399-e2e238d5ee11/volumes" Oct 30 18:17:30.527346 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:17:30.523771 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:17:36.462679 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.462641 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-7bc6687fcf-jpj87"] Oct 30 18:17:36.463642 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.463612 4455 cpu_manager.go:414] "containerMap: removing container" podUID="14a96f10-41c5-44ba-b8ca-86d9fb9520cb" containerName="payload-processing" Oct 30 18:17:36.463642 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.463636 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14a96f10-41c5-44ba-b8ca-86d9fb9520cb" containerName="payload-processing" Oct 30 18:17:36.463813 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.463667 4455 cpu_manager.go:414] "containerMap: removing container" podUID="dcef406f-9a07-4803-b883-aac4b334de0c" containerName="payload-processing" Oct 30 18:17:36.463813 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.463673 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcef406f-9a07-4803-b883-aac4b334de0c" containerName="payload-processing" Oct 30 18:17:36.463813 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.463686 4455 cpu_manager.go:414] "containerMap: removing container" podUID="9bf6d8f4-3ec1-449f-9399-e2e238d5ee11" containerName="maas-api" Oct 30 18:17:36.463813 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.463693 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bf6d8f4-3ec1-449f-9399-e2e238d5ee11" containerName="maas-api" Oct 30 18:17:36.463813 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.463728 4455 cpu_manager.go:414] "containerMap: removing container" podUID="52e2e782-aab7-463d-b91a-e37a774091ab" containerName="maas-api" Oct 30 18:17:36.463813 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.463738 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52e2e782-aab7-463d-b91a-e37a774091ab" containerName="maas-api" Oct 30 18:17:36.463813 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.463757 4455 cpu_manager.go:414] "containerMap: removing container" podUID="9bf6d8f4-3ec1-449f-9399-e2e238d5ee11" containerName="maas-api" Oct 30 18:17:36.463813 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.463762 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bf6d8f4-3ec1-449f-9399-e2e238d5ee11" containerName="maas-api" Oct 30 18:17:36.463813 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.463782 4455 cpu_manager.go:414] "containerMap: removing container" podUID="9bf6d8f4-3ec1-449f-9399-e2e238d5ee11" containerName="maas-api" Oct 30 18:17:36.463813 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.463787 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bf6d8f4-3ec1-449f-9399-e2e238d5ee11" containerName="maas-api" Oct 30 18:17:36.463813 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.463798 4455 cpu_manager.go:414] "containerMap: removing container" podUID="0bd62dee-70b4-4f07-9f41-3186f5550228" containerName="payload-pre-processing" Oct 30 18:17:36.463813 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.463802 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bd62dee-70b4-4f07-9f41-3186f5550228" containerName="payload-pre-processing" Oct 30 18:17:36.464143 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.463906 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcef406f-9a07-4803-b883-aac4b334de0c" containerName="payload-processing" Oct 30 18:17:36.464143 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.463917 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bf6d8f4-3ec1-449f-9399-e2e238d5ee11" containerName="maas-api" Oct 30 18:17:36.464143 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.463923 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="52e2e782-aab7-463d-b91a-e37a774091ab" containerName="maas-api" Oct 30 18:17:36.464143 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.463930 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bf6d8f4-3ec1-449f-9399-e2e238d5ee11" containerName="maas-api" Oct 30 18:17:36.464143 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.463936 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bf6d8f4-3ec1-449f-9399-e2e238d5ee11" containerName="maas-api" Oct 30 18:17:36.464143 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.463943 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bf6d8f4-3ec1-449f-9399-e2e238d5ee11" containerName="maas-api" Oct 30 18:17:36.464143 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.463950 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bd62dee-70b4-4f07-9f41-3186f5550228" containerName="payload-pre-processing" Oct 30 18:17:36.464143 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.463956 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="14a96f10-41c5-44ba-b8ca-86d9fb9520cb" containerName="payload-processing" Oct 30 18:17:36.468994 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.468972 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-7bc6687fcf-jpj87" Oct 30 18:17:36.488768 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.488740 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-7bc6687fcf-jpj87"] Oct 30 18:17:36.497812 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.497786 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-853d3a-dockercfg-5zgxj\"" Oct 30 18:17:36.498975 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.498950 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-853d3a\"" Oct 30 18:17:36.536828 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.536792 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaa3c20b-dc94-49b3-bb4b-3172c5f85879-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-853d3a-7bc6687fcf-jpj87\" (UID: \"aaa3c20b-dc94-49b3-bb4b-3172c5f85879\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-7bc6687fcf-jpj87" Oct 30 18:17:36.537213 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.537181 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqjnj\" (UniqueName: \"kubernetes.io/projected/aaa3c20b-dc94-49b3-bb4b-3172c5f85879-kube-api-access-qqjnj\") pod \"payload-pre-processing-e2e-ipp-a-853d3a-7bc6687fcf-jpj87\" (UID: \"aaa3c20b-dc94-49b3-bb4b-3172c5f85879\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-7bc6687fcf-jpj87" Oct 30 18:17:36.542700 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.542667 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-853d3a-77b4f9b8b4-999hg"] Oct 30 18:17:36.543862 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.543835 4455 cpu_manager.go:414] "containerMap: removing container" podUID="9bf6d8f4-3ec1-449f-9399-e2e238d5ee11" containerName="maas-api" Oct 30 18:17:36.543981 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.543864 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bf6d8f4-3ec1-449f-9399-e2e238d5ee11" containerName="maas-api" Oct 30 18:17:36.548942 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.548907 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-77b4f9b8b4-999hg" Oct 30 18:17:36.575178 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.575124 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-853d3a-77b4f9b8b4-999hg"] Oct 30 18:17:36.660542 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.660502 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmrw6\" (UniqueName: \"kubernetes.io/projected/8d32ebf1-5535-4c0b-bb32-dd07cdc42235-kube-api-access-fmrw6\") pod \"payload-processing-e2e-ipp-a-853d3a-77b4f9b8b4-999hg\" (UID: \"8d32ebf1-5535-4c0b-bb32-dd07cdc42235\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-77b4f9b8b4-999hg" Oct 30 18:17:36.662739 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.662713 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqjnj\" (UniqueName: \"kubernetes.io/projected/aaa3c20b-dc94-49b3-bb4b-3172c5f85879-kube-api-access-qqjnj\") pod \"payload-pre-processing-e2e-ipp-a-853d3a-7bc6687fcf-jpj87\" (UID: \"aaa3c20b-dc94-49b3-bb4b-3172c5f85879\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-7bc6687fcf-jpj87" Oct 30 18:17:36.668511 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.668476 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaa3c20b-dc94-49b3-bb4b-3172c5f85879-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-853d3a-7bc6687fcf-jpj87\" (UID: \"aaa3c20b-dc94-49b3-bb4b-3172c5f85879\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-7bc6687fcf-jpj87" Oct 30 18:17:36.670050 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.669547 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaa3c20b-dc94-49b3-bb4b-3172c5f85879-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-853d3a-7bc6687fcf-jpj87\" (UID: \"aaa3c20b-dc94-49b3-bb4b-3172c5f85879\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-7bc6687fcf-jpj87" Oct 30 18:17:36.670359 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.670206 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d32ebf1-5535-4c0b-bb32-dd07cdc42235-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-853d3a-77b4f9b8b4-999hg\" (UID: \"8d32ebf1-5535-4c0b-bb32-dd07cdc42235\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-77b4f9b8b4-999hg" Oct 30 18:17:36.856391 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.846586 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d32ebf1-5535-4c0b-bb32-dd07cdc42235-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-853d3a-77b4f9b8b4-999hg\" (UID: \"8d32ebf1-5535-4c0b-bb32-dd07cdc42235\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-77b4f9b8b4-999hg" Oct 30 18:17:36.856391 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.846839 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmrw6\" (UniqueName: \"kubernetes.io/projected/8d32ebf1-5535-4c0b-bb32-dd07cdc42235-kube-api-access-fmrw6\") pod \"payload-processing-e2e-ipp-a-853d3a-77b4f9b8b4-999hg\" (UID: \"8d32ebf1-5535-4c0b-bb32-dd07cdc42235\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-77b4f9b8b4-999hg" Oct 30 18:17:36.856391 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.848341 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d32ebf1-5535-4c0b-bb32-dd07cdc42235-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-853d3a-77b4f9b8b4-999hg\" (UID: \"8d32ebf1-5535-4c0b-bb32-dd07cdc42235\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-77b4f9b8b4-999hg" Oct 30 18:17:36.860932 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.860850 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqjnj\" (UniqueName: \"kubernetes.io/projected/aaa3c20b-dc94-49b3-bb4b-3172c5f85879-kube-api-access-qqjnj\") pod \"payload-pre-processing-e2e-ipp-a-853d3a-7bc6687fcf-jpj87\" (UID: \"aaa3c20b-dc94-49b3-bb4b-3172c5f85879\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-7bc6687fcf-jpj87" Oct 30 18:17:36.890747 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.890711 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmrw6\" (UniqueName: \"kubernetes.io/projected/8d32ebf1-5535-4c0b-bb32-dd07cdc42235-kube-api-access-fmrw6\") pod \"payload-processing-e2e-ipp-a-853d3a-77b4f9b8b4-999hg\" (UID: \"8d32ebf1-5535-4c0b-bb32-dd07cdc42235\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-77b4f9b8b4-999hg" Oct 30 18:17:36.991986 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:36.991636 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-77b4f9b8b4-999hg" Oct 30 18:17:37.086148 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:37.083641 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-7bc6687fcf-jpj87" Oct 30 18:17:37.407076 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:17:37.407046 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d32ebf1_5535_4c0b_bb32_dd07cdc42235.slice/crio-c2da7f5df9594333d7245921789f0d4257cd90f3b386e200e8dc7ed230b6d806 WatchSource:0}: Error finding container c2da7f5df9594333d7245921789f0d4257cd90f3b386e200e8dc7ed230b6d806: Status 404 returned error can't find the container with id c2da7f5df9594333d7245921789f0d4257cd90f3b386e200e8dc7ed230b6d806 Oct 30 18:17:37.410859 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:37.410839 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-853d3a-77b4f9b8b4-999hg"] Oct 30 18:17:37.431315 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:37.431295 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-7bc6687fcf-jpj87"] Oct 30 18:17:38.324999 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:38.324964 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-7bc6687fcf-jpj87" event={"ID":"aaa3c20b-dc94-49b3-bb4b-3172c5f85879","Type":"ContainerStarted","Data":"429adae083ca4ce8fa7c2e3eb0776fb1a572d259b27b26d441af119793258a56"} Oct 30 18:17:38.325539 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:38.325032 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-7bc6687fcf-jpj87" Oct 30 18:17:38.325539 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:38.325055 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-7bc6687fcf-jpj87" event={"ID":"aaa3c20b-dc94-49b3-bb4b-3172c5f85879","Type":"ContainerStarted","Data":"6c8495e0daacc79b0e090726c2a6a022c79148382ab33af347fafdfc86010cfa"} Oct 30 18:17:38.326719 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:38.326682 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-77b4f9b8b4-999hg" event={"ID":"8d32ebf1-5535-4c0b-bb32-dd07cdc42235","Type":"ContainerStarted","Data":"3894e4a8c2ea219db5a30edc627f48b2a52f03d83bd2bb00db3ac5257441033a"} Oct 30 18:17:38.326827 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:38.326731 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-77b4f9b8b4-999hg" event={"ID":"8d32ebf1-5535-4c0b-bb32-dd07cdc42235","Type":"ContainerStarted","Data":"c2da7f5df9594333d7245921789f0d4257cd90f3b386e200e8dc7ed230b6d806"} Oct 30 18:17:38.326827 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:38.326755 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-77b4f9b8b4-999hg" Oct 30 18:17:38.371925 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:38.371866 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-7bc6687fcf-jpj87" podStartSLOduration=2.371847842 podStartE2EDuration="2.371847842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:38.368929941 +0000 UTC m=+1510.856566976" watchObservedRunningTime="2026-10-30 18:17:38.371847842 +0000 UTC m=+1510.859484878" Oct 30 18:17:38.419054 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:38.419004 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-77b4f9b8b4-999hg" podStartSLOduration=2.418991061 podStartE2EDuration="2.418991061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:38.412115493 +0000 UTC m=+1510.899752526" watchObservedRunningTime="2026-10-30 18:17:38.418991061 +0000 UTC m=+1510.906628094" Oct 30 18:17:39.621643 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:39.621612 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-75cbbbf795-jcxkn"] Oct 30 18:17:39.625991 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:39.625974 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-75cbbbf795-jcxkn" Oct 30 18:17:39.647165 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:39.647143 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-064408\"" Oct 30 18:17:39.656128 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:39.656107 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-064408-dockercfg-lsnk7\"" Oct 30 18:17:39.732348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:39.730432 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-75cbbbf795-jcxkn"] Oct 30 18:17:39.790442 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:39.789672 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-064408-6c67ff6bd-8tznm"] Oct 30 18:17:39.823396 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:39.802413 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/691071d4-c640-4df5-9882-c58de6741882-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-064408-75cbbbf795-jcxkn\" (UID: \"691071d4-c640-4df5-9882-c58de6741882\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-75cbbbf795-jcxkn" Oct 30 18:17:39.823396 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:39.802519 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s4lt\" (UniqueName: \"kubernetes.io/projected/691071d4-c640-4df5-9882-c58de6741882-kube-api-access-4s4lt\") pod \"payload-pre-processing-e2e-ipp-a-064408-75cbbbf795-jcxkn\" (UID: \"691071d4-c640-4df5-9882-c58de6741882\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-75cbbbf795-jcxkn" Oct 30 18:17:39.915553 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:39.906857 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/691071d4-c640-4df5-9882-c58de6741882-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-064408-75cbbbf795-jcxkn\" (UID: \"691071d4-c640-4df5-9882-c58de6741882\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-75cbbbf795-jcxkn" Oct 30 18:17:39.915553 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:39.906947 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s4lt\" (UniqueName: \"kubernetes.io/projected/691071d4-c640-4df5-9882-c58de6741882-kube-api-access-4s4lt\") pod \"payload-pre-processing-e2e-ipp-a-064408-75cbbbf795-jcxkn\" (UID: \"691071d4-c640-4df5-9882-c58de6741882\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-75cbbbf795-jcxkn" Oct 30 18:17:39.935346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:39.930677 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/691071d4-c640-4df5-9882-c58de6741882-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-064408-75cbbbf795-jcxkn\" (UID: \"691071d4-c640-4df5-9882-c58de6741882\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-75cbbbf795-jcxkn" Oct 30 18:17:39.935346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:39.933762 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-064408-6c67ff6bd-8tznm"] Oct 30 18:17:39.935346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:39.933912 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-6c67ff6bd-8tznm" Oct 30 18:17:39.971490 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:39.970791 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s4lt\" (UniqueName: \"kubernetes.io/projected/691071d4-c640-4df5-9882-c58de6741882-kube-api-access-4s4lt\") pod \"payload-pre-processing-e2e-ipp-a-064408-75cbbbf795-jcxkn\" (UID: \"691071d4-c640-4df5-9882-c58de6741882\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-75cbbbf795-jcxkn" Oct 30 18:17:40.112283 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:40.112245 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-064408-6c67ff6bd-8tznm\" (UID: \"5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-6c67ff6bd-8tznm" Oct 30 18:17:40.112498 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:40.112388 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppbjl\" (UniqueName: \"kubernetes.io/projected/5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5-kube-api-access-ppbjl\") pod \"payload-processing-e2e-ipp-a-064408-6c67ff6bd-8tznm\" (UID: \"5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-6c67ff6bd-8tznm" Oct 30 18:17:40.214537 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:40.214456 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-064408-6c67ff6bd-8tznm\" (UID: \"5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-6c67ff6bd-8tznm" Oct 30 18:17:40.214537 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:40.214520 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppbjl\" (UniqueName: \"kubernetes.io/projected/5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5-kube-api-access-ppbjl\") pod \"payload-processing-e2e-ipp-a-064408-6c67ff6bd-8tznm\" (UID: \"5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-6c67ff6bd-8tznm" Oct 30 18:17:40.215154 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:40.215134 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-064408-6c67ff6bd-8tznm\" (UID: \"5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-6c67ff6bd-8tznm" Oct 30 18:17:40.240202 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:40.240178 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppbjl\" (UniqueName: \"kubernetes.io/projected/5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5-kube-api-access-ppbjl\") pod \"payload-processing-e2e-ipp-a-064408-6c67ff6bd-8tznm\" (UID: \"5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-6c67ff6bd-8tznm" Oct 30 18:17:40.242110 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:40.242078 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-75cbbbf795-jcxkn" Oct 30 18:17:40.310264 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:40.310223 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-6c67ff6bd-8tznm" Oct 30 18:17:40.551165 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:17:40.551132 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod691071d4_c640_4df5_9882_c58de6741882.slice/crio-0960d57cc14db8bb715122be9b973008bf2fbed6402cd9f50f30655fc7c05601 WatchSource:0}: Error finding container 0960d57cc14db8bb715122be9b973008bf2fbed6402cd9f50f30655fc7c05601: Status 404 returned error can't find the container with id 0960d57cc14db8bb715122be9b973008bf2fbed6402cd9f50f30655fc7c05601 Oct 30 18:17:40.553395 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:40.553373 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-75cbbbf795-jcxkn"] Oct 30 18:17:40.641688 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:17:40.641654 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ddda28e_f4e1_4e1b_b229_ebb2b1dd15a5.slice/crio-9f71eac750cf4ce755895b851691715a56c162b834e4555b33b55759838be311 WatchSource:0}: Error finding container 9f71eac750cf4ce755895b851691715a56c162b834e4555b33b55759838be311: Status 404 returned error can't find the container with id 9f71eac750cf4ce755895b851691715a56c162b834e4555b33b55759838be311 Oct 30 18:17:40.644552 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:40.644527 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-064408-6c67ff6bd-8tznm"] Oct 30 18:17:41.356556 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:41.356516 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-6c67ff6bd-8tznm" event={"ID":"5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5","Type":"ContainerStarted","Data":"452e5d2ad27f3778e38cbefb0539323968866e2c23878a0105a5f00bed090124"} Oct 30 18:17:41.356726 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:41.356581 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-6c67ff6bd-8tznm" event={"ID":"5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5","Type":"ContainerStarted","Data":"9f71eac750cf4ce755895b851691715a56c162b834e4555b33b55759838be311"} Oct 30 18:17:41.356726 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:41.356609 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-6c67ff6bd-8tznm" Oct 30 18:17:41.358377 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:41.358349 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-75cbbbf795-jcxkn" event={"ID":"691071d4-c640-4df5-9882-c58de6741882","Type":"ContainerStarted","Data":"2423448c78707ec90c98efe1ea93a5c16833743e632c060190e590ab04cbe01e"} Oct 30 18:17:41.358482 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:41.358396 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-75cbbbf795-jcxkn" event={"ID":"691071d4-c640-4df5-9882-c58de6741882","Type":"ContainerStarted","Data":"0960d57cc14db8bb715122be9b973008bf2fbed6402cd9f50f30655fc7c05601"} Oct 30 18:17:41.358482 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:41.358444 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-75cbbbf795-jcxkn" Oct 30 18:17:41.405210 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:41.405150 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-6c67ff6bd-8tznm" podStartSLOduration=2.405131528 podStartE2EDuration="2.405131528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:41.401224112 +0000 UTC m=+1513.888861146" watchObservedRunningTime="2026-10-30 18:17:41.405131528 +0000 UTC m=+1513.892768564" Oct 30 18:17:41.445022 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:41.444969 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-75cbbbf795-jcxkn" podStartSLOduration=2.444954808 podStartE2EDuration="2.444954808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:41.443829666 +0000 UTC m=+1513.931466702" watchObservedRunningTime="2026-10-30 18:17:41.444954808 +0000 UTC m=+1513.932591841" Oct 30 18:17:41.913948 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:41.913916 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-xmhnq"] Oct 30 18:17:41.918650 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:41.918625 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-xmhnq" Oct 30 18:17:41.937565 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:41.937447 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c6b2120f\"" Oct 30 18:17:41.944054 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:41.944024 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-xmhnq"] Oct 30 18:17:42.041217 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.041182 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c6b2120f-dockercfg-tj5mb\"" Oct 30 18:17:42.130396 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.050356 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72fd9184-e60c-4e6d-8415-e6e146e8b5a2-plugins-config-volume\") pod \"payload-processing-e2e-mt-c6b2120f-5666cb5864-xmhnq\" (UID: \"72fd9184-e60c-4e6d-8415-e6e146e8b5a2\") " pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-xmhnq" Oct 30 18:17:42.130396 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.050712 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsxgw\" (UniqueName: \"kubernetes.io/projected/72fd9184-e60c-4e6d-8415-e6e146e8b5a2-kube-api-access-rsxgw\") pod \"payload-processing-e2e-mt-c6b2120f-5666cb5864-xmhnq\" (UID: \"72fd9184-e60c-4e6d-8415-e6e146e8b5a2\") " pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-xmhnq" Oct 30 18:17:42.171435 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.156157 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsxgw\" (UniqueName: \"kubernetes.io/projected/72fd9184-e60c-4e6d-8415-e6e146e8b5a2-kube-api-access-rsxgw\") pod \"payload-processing-e2e-mt-c6b2120f-5666cb5864-xmhnq\" (UID: \"72fd9184-e60c-4e6d-8415-e6e146e8b5a2\") " pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-xmhnq" Oct 30 18:17:42.249596 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.248518 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72fd9184-e60c-4e6d-8415-e6e146e8b5a2-plugins-config-volume\") pod \"payload-processing-e2e-mt-c6b2120f-5666cb5864-xmhnq\" (UID: \"72fd9184-e60c-4e6d-8415-e6e146e8b5a2\") " pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-xmhnq" Oct 30 18:17:42.251882 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.191425 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsxgw\" (UniqueName: \"kubernetes.io/projected/72fd9184-e60c-4e6d-8415-e6e146e8b5a2-kube-api-access-rsxgw\") pod \"payload-processing-e2e-mt-c6b2120f-5666cb5864-xmhnq\" (UID: \"72fd9184-e60c-4e6d-8415-e6e146e8b5a2\") " pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-xmhnq" Oct 30 18:17:42.265687 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.254084 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72fd9184-e60c-4e6d-8415-e6e146e8b5a2-plugins-config-volume\") pod \"payload-processing-e2e-mt-c6b2120f-5666cb5864-xmhnq\" (UID: \"72fd9184-e60c-4e6d-8415-e6e146e8b5a2\") " pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-xmhnq" Oct 30 18:17:42.265687 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.254473 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-xmhnq" Oct 30 18:17:42.278974 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.278674 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-8hk9k"] Oct 30 18:17:42.285187 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.285160 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-8hk9k"] Oct 30 18:17:42.285393 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.285267 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-8hk9k" Oct 30 18:17:42.451869 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.451765 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3357893d-bb50-423d-95f8-2b7c3bd7dc5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-8hk9k\" (UID: \"3357893d-bb50-423d-95f8-2b7c3bd7dc5e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-8hk9k" Oct 30 18:17:42.452031 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.451926 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvn2h\" (UniqueName: \"kubernetes.io/projected/3357893d-bb50-423d-95f8-2b7c3bd7dc5e-kube-api-access-pvn2h\") pod \"payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-8hk9k\" (UID: \"3357893d-bb50-423d-95f8-2b7c3bd7dc5e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-8hk9k" Oct 30 18:17:42.553528 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.553493 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvn2h\" (UniqueName: \"kubernetes.io/projected/3357893d-bb50-423d-95f8-2b7c3bd7dc5e-kube-api-access-pvn2h\") pod \"payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-8hk9k\" (UID: \"3357893d-bb50-423d-95f8-2b7c3bd7dc5e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-8hk9k" Oct 30 18:17:42.553839 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.553819 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3357893d-bb50-423d-95f8-2b7c3bd7dc5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-8hk9k\" (UID: \"3357893d-bb50-423d-95f8-2b7c3bd7dc5e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-8hk9k" Oct 30 18:17:42.554588 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.554564 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3357893d-bb50-423d-95f8-2b7c3bd7dc5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-8hk9k\" (UID: \"3357893d-bb50-423d-95f8-2b7c3bd7dc5e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-8hk9k" Oct 30 18:17:42.579786 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.579592 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvn2h\" (UniqueName: \"kubernetes.io/projected/3357893d-bb50-423d-95f8-2b7c3bd7dc5e-kube-api-access-pvn2h\") pod \"payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-8hk9k\" (UID: \"3357893d-bb50-423d-95f8-2b7c3bd7dc5e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-8hk9k" Oct 30 18:17:42.582182 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.581704 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9"] Oct 30 18:17:42.588586 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.588561 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9" Oct 30 18:17:42.592536 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.592508 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-xmhnq"] Oct 30 18:17:42.594663 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:17:42.594626 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72fd9184_e60c_4e6d_8415_e6e146e8b5a2.slice/crio-54685b6ce40e4d6c33cc2f46ef2f3f87a3d71e64676b80faf5482e80a426c2d8 WatchSource:0}: Error finding container 54685b6ce40e4d6c33cc2f46ef2f3f87a3d71e64676b80faf5482e80a426c2d8: Status 404 returned error can't find the container with id 54685b6ce40e4d6c33cc2f46ef2f3f87a3d71e64676b80faf5482e80a426c2d8 Oct 30 18:17:42.606448 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.606426 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c6b2120f-serving-cert\"" Oct 30 18:17:42.625099 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.624697 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9"] Oct 30 18:17:42.643513 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.643483 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-8hk9k" Oct 30 18:17:42.813424 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.777339 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd1096fc-1f87-4e00-b6ec-2195cb6be82a-metrics-certs\") pod \"maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9\" (UID: \"fd1096fc-1f87-4e00-b6ec-2195cb6be82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9" Oct 30 18:17:42.813424 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.777853 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd1096fc-1f87-4e00-b6ec-2195cb6be82a-maas-api-tls\") pod \"maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9\" (UID: \"fd1096fc-1f87-4e00-b6ec-2195cb6be82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9" Oct 30 18:17:42.813424 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.778193 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkz97\" (UniqueName: \"kubernetes.io/projected/fd1096fc-1f87-4e00-b6ec-2195cb6be82a-kube-api-access-tkz97\") pod \"maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9\" (UID: \"fd1096fc-1f87-4e00-b6ec-2195cb6be82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9" Oct 30 18:17:42.813424 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.778291 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1096fc-1f87-4e00-b6ec-2195cb6be82a-openshift-service-ca\") pod \"maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9\" (UID: \"fd1096fc-1f87-4e00-b6ec-2195cb6be82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9" Oct 30 18:17:42.887439 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.880444 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkz97\" (UniqueName: \"kubernetes.io/projected/fd1096fc-1f87-4e00-b6ec-2195cb6be82a-kube-api-access-tkz97\") pod \"maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9\" (UID: \"fd1096fc-1f87-4e00-b6ec-2195cb6be82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9" Oct 30 18:17:42.887439 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.880546 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1096fc-1f87-4e00-b6ec-2195cb6be82a-openshift-service-ca\") pod \"maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9\" (UID: \"fd1096fc-1f87-4e00-b6ec-2195cb6be82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9" Oct 30 18:17:42.887439 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.880654 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd1096fc-1f87-4e00-b6ec-2195cb6be82a-metrics-certs\") pod \"maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9\" (UID: \"fd1096fc-1f87-4e00-b6ec-2195cb6be82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9" Oct 30 18:17:42.887439 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.880824 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd1096fc-1f87-4e00-b6ec-2195cb6be82a-maas-api-tls\") pod \"maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9\" (UID: \"fd1096fc-1f87-4e00-b6ec-2195cb6be82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9" Oct 30 18:17:42.887439 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.882939 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1096fc-1f87-4e00-b6ec-2195cb6be82a-openshift-service-ca\") pod \"maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9\" (UID: \"fd1096fc-1f87-4e00-b6ec-2195cb6be82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9" Oct 30 18:17:42.887439 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.884250 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd1096fc-1f87-4e00-b6ec-2195cb6be82a-maas-api-tls\") pod \"maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9\" (UID: \"fd1096fc-1f87-4e00-b6ec-2195cb6be82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9" Oct 30 18:17:42.887439 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.885595 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd1096fc-1f87-4e00-b6ec-2195cb6be82a-metrics-certs\") pod \"maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9\" (UID: \"fd1096fc-1f87-4e00-b6ec-2195cb6be82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9" Oct 30 18:17:42.909712 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.909682 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkz97\" (UniqueName: \"kubernetes.io/projected/fd1096fc-1f87-4e00-b6ec-2195cb6be82a-kube-api-access-tkz97\") pod \"maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9\" (UID: \"fd1096fc-1f87-4e00-b6ec-2195cb6be82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9" Oct 30 18:17:42.956732 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.956697 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9" Oct 30 18:17:42.965655 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:42.965629 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-8hk9k"] Oct 30 18:17:42.972435 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:17:42.971225 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3357893d_bb50_423d_95f8_2b7c3bd7dc5e.slice/crio-4e8880f0cba7b92647d2def17a1489ab750693322006b783811a277efa6654a9 WatchSource:0}: Error finding container 4e8880f0cba7b92647d2def17a1489ab750693322006b783811a277efa6654a9: Status 404 returned error can't find the container with id 4e8880f0cba7b92647d2def17a1489ab750693322006b783811a277efa6654a9 Oct 30 18:17:43.286886 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:17:43.285625 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd1096fc_1f87_4e00_b6ec_2195cb6be82a.slice/crio-a3766fc14ef49903c4866d5bc4bbf248393a74173c8768851096306ebd0b177e WatchSource:0}: Error finding container a3766fc14ef49903c4866d5bc4bbf248393a74173c8768851096306ebd0b177e: Status 404 returned error can't find the container with id a3766fc14ef49903c4866d5bc4bbf248393a74173c8768851096306ebd0b177e Oct 30 18:17:43.287653 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:43.287632 4455 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:17:43.287823 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:43.287805 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9"] Oct 30 18:17:43.374928 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:43.374888 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-8hk9k" event={"ID":"3357893d-bb50-423d-95f8-2b7c3bd7dc5e","Type":"ContainerStarted","Data":"927022c133242a12b425b739630114283a31396c1abdefce516c0bf7e44036cd"} Oct 30 18:17:43.375126 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:43.374958 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-8hk9k" event={"ID":"3357893d-bb50-423d-95f8-2b7c3bd7dc5e","Type":"ContainerStarted","Data":"4e8880f0cba7b92647d2def17a1489ab750693322006b783811a277efa6654a9"} Oct 30 18:17:43.375207 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:43.375124 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-8hk9k" Oct 30 18:17:43.376472 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:43.376446 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9" event={"ID":"fd1096fc-1f87-4e00-b6ec-2195cb6be82a","Type":"ContainerStarted","Data":"a3766fc14ef49903c4866d5bc4bbf248393a74173c8768851096306ebd0b177e"} Oct 30 18:17:43.378300 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:43.378275 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-xmhnq" event={"ID":"72fd9184-e60c-4e6d-8415-e6e146e8b5a2","Type":"ContainerStarted","Data":"3b121fdd61d3d5d5aa2e6d9477b94911d97a9644a1426897d9efa03c396d2929"} Oct 30 18:17:43.378493 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:43.378335 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-xmhnq" event={"ID":"72fd9184-e60c-4e6d-8415-e6e146e8b5a2","Type":"ContainerStarted","Data":"54685b6ce40e4d6c33cc2f46ef2f3f87a3d71e64676b80faf5482e80a426c2d8"} Oct 30 18:17:43.378493 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:43.378373 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-xmhnq" Oct 30 18:17:43.422476 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:43.422411 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-8hk9k" podStartSLOduration=2.42239851 podStartE2EDuration="2.42239851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:43.418284912 +0000 UTC m=+1515.905921959" watchObservedRunningTime="2026-10-30 18:17:43.42239851 +0000 UTC m=+1515.910035544" Oct 30 18:17:44.387968 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:44.387931 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9" event={"ID":"fd1096fc-1f87-4e00-b6ec-2195cb6be82a","Type":"ContainerStarted","Data":"2d392a5e47fb00522c705813e1e71ae6c9bc151d03e82e991b45324ca3df6f77"} Oct 30 18:17:44.388543 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:44.388010 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9" Oct 30 18:17:44.472058 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:44.471998 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9" podStartSLOduration=1.8441577900000001 podStartE2EDuration="2.471978619s" totalImagesPullingTime="627.820829ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:44.46749276 +0000 UTC m=+1516.955129811" watchObservedRunningTime="2026-10-30 18:17:44.471978619 +0000 UTC m=+1516.959615667" Oct 30 18:17:44.473721 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:44.473674 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-xmhnq" podStartSLOduration=3.473662533 podStartE2EDuration="3.473662533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:43.474856194 +0000 UTC m=+1515.962493228" watchObservedRunningTime="2026-10-30 18:17:44.473662533 +0000 UTC m=+1516.961299567" Oct 30 18:17:48.615651 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:48.615616 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-da151d-697b678888-d8zsf"] Oct 30 18:17:48.653777 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:48.653750 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-da151d-6cfcff59d7-rqzfd"] Oct 30 18:17:49.365477 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:49.359257 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-77b4f9b8b4-999hg" Oct 30 18:17:49.365477 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:49.361484 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-7bc6687fcf-jpj87" Oct 30 18:17:49.706405 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:49.703161 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t"] Oct 30 18:17:49.729409 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:49.729376 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t" Oct 30 18:17:49.734792 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:49.734752 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t"] Oct 30 18:17:49.746045 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:49.746015 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-853d3a-serving-cert\"" Oct 30 18:17:49.798076 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:49.796897 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/347e628a-8625-4f90-8061-8f24241182ee-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t\" (UID: \"347e628a-8625-4f90-8061-8f24241182ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t" Oct 30 18:17:49.798076 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:49.796969 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/347e628a-8625-4f90-8061-8f24241182ee-maas-api-tls\") pod \"maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t\" (UID: \"347e628a-8625-4f90-8061-8f24241182ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t" Oct 30 18:17:49.798076 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:49.797013 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/347e628a-8625-4f90-8061-8f24241182ee-metrics-certs\") pod \"maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t\" (UID: \"347e628a-8625-4f90-8061-8f24241182ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t" Oct 30 18:17:49.798076 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:49.797149 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plctx\" (UniqueName: \"kubernetes.io/projected/347e628a-8625-4f90-8061-8f24241182ee-kube-api-access-plctx\") pod \"maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t\" (UID: \"347e628a-8625-4f90-8061-8f24241182ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t" Oct 30 18:17:49.900569 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:49.900532 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/347e628a-8625-4f90-8061-8f24241182ee-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t\" (UID: \"347e628a-8625-4f90-8061-8f24241182ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t" Oct 30 18:17:49.901035 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:49.901008 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/347e628a-8625-4f90-8061-8f24241182ee-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t\" (UID: \"347e628a-8625-4f90-8061-8f24241182ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t" Oct 30 18:17:49.901264 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:49.901243 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/347e628a-8625-4f90-8061-8f24241182ee-maas-api-tls\") pod \"maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t\" (UID: \"347e628a-8625-4f90-8061-8f24241182ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t" Oct 30 18:17:49.901893 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:49.901874 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/347e628a-8625-4f90-8061-8f24241182ee-metrics-certs\") pod \"maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t\" (UID: \"347e628a-8625-4f90-8061-8f24241182ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t" Oct 30 18:17:49.902602 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:49.902582 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plctx\" (UniqueName: \"kubernetes.io/projected/347e628a-8625-4f90-8061-8f24241182ee-kube-api-access-plctx\") pod \"maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t\" (UID: \"347e628a-8625-4f90-8061-8f24241182ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t" Oct 30 18:17:49.921292 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:49.921247 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/347e628a-8625-4f90-8061-8f24241182ee-maas-api-tls\") pod \"maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t\" (UID: \"347e628a-8625-4f90-8061-8f24241182ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t" Oct 30 18:17:49.937699 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:49.936726 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/347e628a-8625-4f90-8061-8f24241182ee-metrics-certs\") pod \"maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t\" (UID: \"347e628a-8625-4f90-8061-8f24241182ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t" Oct 30 18:17:49.946437 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:49.946367 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plctx\" (UniqueName: \"kubernetes.io/projected/347e628a-8625-4f90-8061-8f24241182ee-kube-api-access-plctx\") pod \"maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t\" (UID: \"347e628a-8625-4f90-8061-8f24241182ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t" Oct 30 18:17:50.055668 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:50.055627 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t" Oct 30 18:17:50.378608 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:17:50.378561 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod347e628a_8625_4f90_8061_8f24241182ee.slice/crio-d69f2fea0a8d495c5256d1790e15cb7e63566eb3f3f70f880dd77f637b7e25be WatchSource:0}: Error finding container d69f2fea0a8d495c5256d1790e15cb7e63566eb3f3f70f880dd77f637b7e25be: Status 404 returned error can't find the container with id d69f2fea0a8d495c5256d1790e15cb7e63566eb3f3f70f880dd77f637b7e25be Oct 30 18:17:50.380131 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:50.380109 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t"] Oct 30 18:17:50.400992 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:50.400967 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9" Oct 30 18:17:50.514688 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:50.514647 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t" event={"ID":"347e628a-8625-4f90-8061-8f24241182ee","Type":"ContainerStarted","Data":"d69f2fea0a8d495c5256d1790e15cb7e63566eb3f3f70f880dd77f637b7e25be"} Oct 30 18:17:51.523952 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:51.523918 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t" event={"ID":"347e628a-8625-4f90-8061-8f24241182ee","Type":"ContainerStarted","Data":"acd52034a06c0e62e080314e36a3f140eb9c939b7e8ebfa1ba06b8387db75748"} Oct 30 18:17:51.524390 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:51.524016 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t" Oct 30 18:17:51.575246 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:51.575175 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t" podStartSLOduration=2.26586092 podStartE2EDuration="2.575152029s" totalImagesPullingTime="309.291109ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:51.572933232 +0000 UTC m=+1524.060570315" watchObservedRunningTime="2026-10-30 18:17:51.575152029 +0000 UTC m=+1524.062789063" Oct 30 18:17:52.368383 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:52.368351 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-75cbbbf795-jcxkn" Oct 30 18:17:52.368603 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:52.368471 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-6c67ff6bd-8tznm" Oct 30 18:17:53.080076 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:53.080043 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z"] Oct 30 18:17:53.085074 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:53.085051 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z" Oct 30 18:17:53.117590 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:53.117546 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z"] Oct 30 18:17:53.144911 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:53.144875 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-064408-serving-cert\"" Oct 30 18:17:53.235998 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:53.235967 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/916bd30a-6453-477b-9675-8356aea0eaba-metrics-certs\") pod \"maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z\" (UID: \"916bd30a-6453-477b-9675-8356aea0eaba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z" Oct 30 18:17:53.236171 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:53.236035 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/916bd30a-6453-477b-9675-8356aea0eaba-maas-api-tls\") pod \"maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z\" (UID: \"916bd30a-6453-477b-9675-8356aea0eaba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z" Oct 30 18:17:53.236243 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:53.236209 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5d6t\" (UniqueName: \"kubernetes.io/projected/916bd30a-6453-477b-9675-8356aea0eaba-kube-api-access-h5d6t\") pod \"maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z\" (UID: \"916bd30a-6453-477b-9675-8356aea0eaba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z" Oct 30 18:17:53.236290 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:53.236254 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/916bd30a-6453-477b-9675-8356aea0eaba-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z\" (UID: \"916bd30a-6453-477b-9675-8356aea0eaba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z" Oct 30 18:17:53.343759 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:53.339407 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/916bd30a-6453-477b-9675-8356aea0eaba-maas-api-tls\") pod \"maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z\" (UID: \"916bd30a-6453-477b-9675-8356aea0eaba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z" Oct 30 18:17:53.343759 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:53.339614 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5d6t\" (UniqueName: \"kubernetes.io/projected/916bd30a-6453-477b-9675-8356aea0eaba-kube-api-access-h5d6t\") pod \"maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z\" (UID: \"916bd30a-6453-477b-9675-8356aea0eaba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z" Oct 30 18:17:53.343759 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:53.339642 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/916bd30a-6453-477b-9675-8356aea0eaba-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z\" (UID: \"916bd30a-6453-477b-9675-8356aea0eaba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z" Oct 30 18:17:53.343759 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:53.339848 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/916bd30a-6453-477b-9675-8356aea0eaba-metrics-certs\") pod \"maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z\" (UID: \"916bd30a-6453-477b-9675-8356aea0eaba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z" Oct 30 18:17:53.344177 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:53.344145 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/916bd30a-6453-477b-9675-8356aea0eaba-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z\" (UID: \"916bd30a-6453-477b-9675-8356aea0eaba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z" Oct 30 18:17:53.345858 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:53.345820 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/916bd30a-6453-477b-9675-8356aea0eaba-maas-api-tls\") pod \"maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z\" (UID: \"916bd30a-6453-477b-9675-8356aea0eaba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z" Oct 30 18:17:53.347186 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:53.347147 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/916bd30a-6453-477b-9675-8356aea0eaba-metrics-certs\") pod \"maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z\" (UID: \"916bd30a-6453-477b-9675-8356aea0eaba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z" Oct 30 18:17:53.372805 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:53.372743 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5d6t\" (UniqueName: \"kubernetes.io/projected/916bd30a-6453-477b-9675-8356aea0eaba-kube-api-access-h5d6t\") pod \"maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z\" (UID: \"916bd30a-6453-477b-9675-8356aea0eaba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z" Oct 30 18:17:53.416097 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:53.410662 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z" Oct 30 18:17:53.616027 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:53.615932 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-da151d-697b678888-d8zsf" podUID="f62dc6ab-f96a-49ed-9f75-158f121781d7" containerName="payload-processing" containerID="cri-o://f8140bc47e28407013a8f2ca46ee3dff968a1e14218563cc74f095d5bfc4ac04" gracePeriod=25 Oct 30 18:17:53.655005 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:53.654949 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da151d-6cfcff59d7-rqzfd" podUID="ebc64c0e-74bd-4130-bf28-65c0e73f2197" containerName="payload-pre-processing" containerID="cri-o://b27a1305268b2e2c1635e77f8e5c74c987b890dc479600e2ccd0a0472e8906f7" gracePeriod=25 Oct 30 18:17:53.724764 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:17:53.724734 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod916bd30a_6453_477b_9675_8356aea0eaba.slice/crio-1170bcfbaad01d5e453ad3baec7cafdbd07864fb343708e7f3a109d18bd50460 WatchSource:0}: Error finding container 1170bcfbaad01d5e453ad3baec7cafdbd07864fb343708e7f3a109d18bd50460: Status 404 returned error can't find the container with id 1170bcfbaad01d5e453ad3baec7cafdbd07864fb343708e7f3a109d18bd50460 Oct 30 18:17:53.724764 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:53.724768 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z"] Oct 30 18:17:53.997800 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:53.997379 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-da151d-697b678888-d8zsf" Oct 30 18:17:54.081530 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.081498 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da151d-6cfcff59d7-rqzfd" Oct 30 18:17:54.175827 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.175794 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebc64c0e-74bd-4130-bf28-65c0e73f2197-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc64c0e-74bd-4130-bf28-65c0e73f2197-plugins-config-volume\") pod \"ebc64c0e-74bd-4130-bf28-65c0e73f2197\" (UID: \"ebc64c0e-74bd-4130-bf28-65c0e73f2197\") " Oct 30 18:17:54.175984 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.175921 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebc64c0e-74bd-4130-bf28-65c0e73f2197-kube-api-access-n472l\" (UniqueName: \"kubernetes.io/projected/ebc64c0e-74bd-4130-bf28-65c0e73f2197-kube-api-access-n472l\") pod \"ebc64c0e-74bd-4130-bf28-65c0e73f2197\" (UID: \"ebc64c0e-74bd-4130-bf28-65c0e73f2197\") " Oct 30 18:17:54.176540 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.176505 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebc64c0e-74bd-4130-bf28-65c0e73f2197-plugins-config-volume" pod "ebc64c0e-74bd-4130-bf28-65c0e73f2197" (UID: "ebc64c0e-74bd-4130-bf28-65c0e73f2197"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:17:54.176667 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.176598 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f62dc6ab-f96a-49ed-9f75-158f121781d7-kube-api-access-htzfd\" (UniqueName: \"kubernetes.io/projected/f62dc6ab-f96a-49ed-9f75-158f121781d7-kube-api-access-htzfd\") pod \"f62dc6ab-f96a-49ed-9f75-158f121781d7\" (UID: \"f62dc6ab-f96a-49ed-9f75-158f121781d7\") " Oct 30 18:17:54.176768 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.176733 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f62dc6ab-f96a-49ed-9f75-158f121781d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f62dc6ab-f96a-49ed-9f75-158f121781d7-plugins-config-volume\") pod \"f62dc6ab-f96a-49ed-9f75-158f121781d7\" (UID: \"f62dc6ab-f96a-49ed-9f75-158f121781d7\") " Oct 30 18:17:54.177229 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.177202 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f62dc6ab-f96a-49ed-9f75-158f121781d7-plugins-config-volume" pod "f62dc6ab-f96a-49ed-9f75-158f121781d7" (UID: "f62dc6ab-f96a-49ed-9f75-158f121781d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:17:54.177994 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.177968 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f62dc6ab-f96a-49ed-9f75-158f121781d7-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:17:54.177994 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.178000 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc64c0e-74bd-4130-bf28-65c0e73f2197-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:17:54.181768 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.181742 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebc64c0e-74bd-4130-bf28-65c0e73f2197-kube-api-access-n472l" pod "ebc64c0e-74bd-4130-bf28-65c0e73f2197" (UID: "ebc64c0e-74bd-4130-bf28-65c0e73f2197"). InnerVolumeSpecName "kube-api-access-n472l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:54.184367 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.183690 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f62dc6ab-f96a-49ed-9f75-158f121781d7-kube-api-access-htzfd" pod "f62dc6ab-f96a-49ed-9f75-158f121781d7" (UID: "f62dc6ab-f96a-49ed-9f75-158f121781d7"). InnerVolumeSpecName "kube-api-access-htzfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:54.327087 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.325909 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n472l\" (UniqueName: \"kubernetes.io/projected/ebc64c0e-74bd-4130-bf28-65c0e73f2197-kube-api-access-n472l\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:17:54.327087 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.325942 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htzfd\" (UniqueName: \"kubernetes.io/projected/f62dc6ab-f96a-49ed-9f75-158f121781d7-kube-api-access-htzfd\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:17:54.424435 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.424355 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-8hk9k" Oct 30 18:17:54.425551 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.425536 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-xmhnq" Oct 30 18:17:54.553139 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.552447 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z" event={"ID":"916bd30a-6453-477b-9675-8356aea0eaba","Type":"ContainerStarted","Data":"9242162d8e5593f821b02df1b897a5a4885928c5d63a11b487902ef5262de0a4"} Oct 30 18:17:54.553139 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.552897 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z" event={"ID":"916bd30a-6453-477b-9675-8356aea0eaba","Type":"ContainerStarted","Data":"1170bcfbaad01d5e453ad3baec7cafdbd07864fb343708e7f3a109d18bd50460"} Oct 30 18:17:54.553139 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.552927 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z" Oct 30 18:17:54.562356 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.555337 4455 generic.go:356] "Generic (PLEG): container finished" podID="f62dc6ab-f96a-49ed-9f75-158f121781d7" containerID="f8140bc47e28407013a8f2ca46ee3dff968a1e14218563cc74f095d5bfc4ac04" exitCode=0 Oct 30 18:17:54.562356 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.555407 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-da151d-697b678888-d8zsf" Oct 30 18:17:54.562356 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.555476 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-da151d-697b678888-d8zsf" event={"ID":"f62dc6ab-f96a-49ed-9f75-158f121781d7","Type":"ContainerDied","Data":"f8140bc47e28407013a8f2ca46ee3dff968a1e14218563cc74f095d5bfc4ac04"} Oct 30 18:17:54.562356 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.555517 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-da151d-697b678888-d8zsf" event={"ID":"f62dc6ab-f96a-49ed-9f75-158f121781d7","Type":"ContainerDied","Data":"85e2b50dc55df768ff348eb6ce2e161d0c7e9c75b8f742fb786ba60dc90e00b4"} Oct 30 18:17:54.562356 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.555554 4455 scope.go:122] "RemoveContainer" containerID="f8140bc47e28407013a8f2ca46ee3dff968a1e14218563cc74f095d5bfc4ac04" Oct 30 18:17:54.571434 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.568176 4455 generic.go:356] "Generic (PLEG): container finished" podID="ebc64c0e-74bd-4130-bf28-65c0e73f2197" containerID="b27a1305268b2e2c1635e77f8e5c74c987b890dc479600e2ccd0a0472e8906f7" exitCode=0 Oct 30 18:17:54.571434 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.568229 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da151d-6cfcff59d7-rqzfd" event={"ID":"ebc64c0e-74bd-4130-bf28-65c0e73f2197","Type":"ContainerDied","Data":"b27a1305268b2e2c1635e77f8e5c74c987b890dc479600e2ccd0a0472e8906f7"} Oct 30 18:17:54.571434 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.568279 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da151d-6cfcff59d7-rqzfd" event={"ID":"ebc64c0e-74bd-4130-bf28-65c0e73f2197","Type":"ContainerDied","Data":"68a0380a66e2dbf9c9024b183840a3318515a7da47cc5baafbdb4b07531491ef"} Oct 30 18:17:54.571434 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.568376 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da151d-6cfcff59d7-rqzfd" Oct 30 18:17:54.593803 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.593781 4455 scope.go:122] "RemoveContainer" containerID="f8140bc47e28407013a8f2ca46ee3dff968a1e14218563cc74f095d5bfc4ac04" Oct 30 18:17:54.594147 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:17:54.594122 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8140bc47e28407013a8f2ca46ee3dff968a1e14218563cc74f095d5bfc4ac04\": container with ID starting with f8140bc47e28407013a8f2ca46ee3dff968a1e14218563cc74f095d5bfc4ac04 not found: ID does not exist" containerID="f8140bc47e28407013a8f2ca46ee3dff968a1e14218563cc74f095d5bfc4ac04" Oct 30 18:17:54.594270 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.594152 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8140bc47e28407013a8f2ca46ee3dff968a1e14218563cc74f095d5bfc4ac04"} err="failed to get container status \"f8140bc47e28407013a8f2ca46ee3dff968a1e14218563cc74f095d5bfc4ac04\": rpc error: code = NotFound desc = could not find container \"f8140bc47e28407013a8f2ca46ee3dff968a1e14218563cc74f095d5bfc4ac04\": container with ID starting with f8140bc47e28407013a8f2ca46ee3dff968a1e14218563cc74f095d5bfc4ac04 not found: ID does not exist" Oct 30 18:17:54.594270 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.594176 4455 scope.go:122] "RemoveContainer" containerID="b27a1305268b2e2c1635e77f8e5c74c987b890dc479600e2ccd0a0472e8906f7" Oct 30 18:17:54.603020 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.602975 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z" podStartSLOduration=1.256970564 podStartE2EDuration="1.602962846s" totalImagesPullingTime="345.992282ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:54.602200963 +0000 UTC m=+1527.089837999" watchObservedRunningTime="2026-10-30 18:17:54.602962846 +0000 UTC m=+1527.090599880" Oct 30 18:17:54.608718 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.608699 4455 scope.go:122] "RemoveContainer" containerID="b27a1305268b2e2c1635e77f8e5c74c987b890dc479600e2ccd0a0472e8906f7" Oct 30 18:17:54.609033 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:17:54.609011 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b27a1305268b2e2c1635e77f8e5c74c987b890dc479600e2ccd0a0472e8906f7\": container with ID starting with b27a1305268b2e2c1635e77f8e5c74c987b890dc479600e2ccd0a0472e8906f7 not found: ID does not exist" containerID="b27a1305268b2e2c1635e77f8e5c74c987b890dc479600e2ccd0a0472e8906f7" Oct 30 18:17:54.609084 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.609042 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b27a1305268b2e2c1635e77f8e5c74c987b890dc479600e2ccd0a0472e8906f7"} err="failed to get container status \"b27a1305268b2e2c1635e77f8e5c74c987b890dc479600e2ccd0a0472e8906f7\": rpc error: code = NotFound desc = could not find container \"b27a1305268b2e2c1635e77f8e5c74c987b890dc479600e2ccd0a0472e8906f7\": container with ID starting with b27a1305268b2e2c1635e77f8e5c74c987b890dc479600e2ccd0a0472e8906f7 not found: ID does not exist" Oct 30 18:17:54.647104 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.647068 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-da151d-6cfcff59d7-rqzfd"] Oct 30 18:17:54.652009 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.651979 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-da151d-6cfcff59d7-rqzfd"] Oct 30 18:17:54.687136 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.687047 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-da151d-697b678888-d8zsf"] Oct 30 18:17:54.693301 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:54.693268 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-da151d-697b678888-d8zsf"] Oct 30 18:17:55.638617 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:55.638584 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebc64c0e-74bd-4130-bf28-65c0e73f2197" path="/var/lib/kubelet/pods/ebc64c0e-74bd-4130-bf28-65c0e73f2197/volumes" Oct 30 18:17:55.639075 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:55.639061 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f62dc6ab-f96a-49ed-9f75-158f121781d7" path="/var/lib/kubelet/pods/f62dc6ab-f96a-49ed-9f75-158f121781d7/volumes" Oct 30 18:17:57.537935 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:17:57.537906 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t" Oct 30 18:18:00.391456 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:18:00.391413 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:18:00.585196 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:00.585163 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z" Oct 30 18:18:10.097866 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:10.097827 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-853d3a-77b4f9b8b4-999hg"] Oct 30 18:18:10.134178 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:10.134137 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-7bc6687fcf-jpj87"] Oct 30 18:18:13.289856 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:13.289822 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-064408-6c67ff6bd-8tznm"] Oct 30 18:18:13.309158 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:13.309121 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-75cbbbf795-jcxkn"] Oct 30 18:18:15.117422 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:15.108736 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-77b4f9b8b4-999hg" podUID="8d32ebf1-5535-4c0b-bb32-dd07cdc42235" containerName="payload-processing" containerID="cri-o://3894e4a8c2ea219db5a30edc627f48b2a52f03d83bd2bb00db3ac5257441033a" gracePeriod=25 Oct 30 18:18:15.156500 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:15.135098 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-7bc6687fcf-jpj87" podUID="aaa3c20b-dc94-49b3-bb4b-3172c5f85879" containerName="payload-pre-processing" containerID="cri-o://429adae083ca4ce8fa7c2e3eb0776fb1a572d259b27b26d441af119793258a56" gracePeriod=25 Oct 30 18:18:15.508036 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:15.508011 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-77b4f9b8b4-999hg" Oct 30 18:18:15.512095 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:15.512071 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-7bc6687fcf-jpj87" Oct 30 18:18:15.617343 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:15.617287 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d32ebf1-5535-4c0b-bb32-dd07cdc42235-kube-api-access-fmrw6\" (UniqueName: \"kubernetes.io/projected/8d32ebf1-5535-4c0b-bb32-dd07cdc42235-kube-api-access-fmrw6\") pod \"8d32ebf1-5535-4c0b-bb32-dd07cdc42235\" (UID: \"8d32ebf1-5535-4c0b-bb32-dd07cdc42235\") " Oct 30 18:18:15.617631 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:15.617610 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d32ebf1-5535-4c0b-bb32-dd07cdc42235-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d32ebf1-5535-4c0b-bb32-dd07cdc42235-plugins-config-volume\") pod \"8d32ebf1-5535-4c0b-bb32-dd07cdc42235\" (UID: \"8d32ebf1-5535-4c0b-bb32-dd07cdc42235\") " Oct 30 18:18:15.617746 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:15.617685 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaa3c20b-dc94-49b3-bb4b-3172c5f85879-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaa3c20b-dc94-49b3-bb4b-3172c5f85879-plugins-config-volume\") pod \"aaa3c20b-dc94-49b3-bb4b-3172c5f85879\" (UID: \"aaa3c20b-dc94-49b3-bb4b-3172c5f85879\") " Oct 30 18:18:15.617814 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:15.617756 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaa3c20b-dc94-49b3-bb4b-3172c5f85879-kube-api-access-qqjnj\" (UniqueName: \"kubernetes.io/projected/aaa3c20b-dc94-49b3-bb4b-3172c5f85879-kube-api-access-qqjnj\") pod \"aaa3c20b-dc94-49b3-bb4b-3172c5f85879\" (UID: \"aaa3c20b-dc94-49b3-bb4b-3172c5f85879\") " Oct 30 18:18:15.618288 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:15.618250 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d32ebf1-5535-4c0b-bb32-dd07cdc42235-plugins-config-volume" pod "8d32ebf1-5535-4c0b-bb32-dd07cdc42235" (UID: "8d32ebf1-5535-4c0b-bb32-dd07cdc42235"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:15.618517 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:15.618488 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d32ebf1-5535-4c0b-bb32-dd07cdc42235-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:18:15.619016 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:15.618987 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa3c20b-dc94-49b3-bb4b-3172c5f85879-plugins-config-volume" pod "aaa3c20b-dc94-49b3-bb4b-3172c5f85879" (UID: "aaa3c20b-dc94-49b3-bb4b-3172c5f85879"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:15.620443 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:15.620416 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d32ebf1-5535-4c0b-bb32-dd07cdc42235-kube-api-access-fmrw6" pod "8d32ebf1-5535-4c0b-bb32-dd07cdc42235" (UID: "8d32ebf1-5535-4c0b-bb32-dd07cdc42235"). InnerVolumeSpecName "kube-api-access-fmrw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:15.621393 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:15.621368 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaa3c20b-dc94-49b3-bb4b-3172c5f85879-kube-api-access-qqjnj" pod "aaa3c20b-dc94-49b3-bb4b-3172c5f85879" (UID: "aaa3c20b-dc94-49b3-bb4b-3172c5f85879"). InnerVolumeSpecName "kube-api-access-qqjnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:15.719919 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:15.719890 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaa3c20b-dc94-49b3-bb4b-3172c5f85879-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:18:15.719919 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:15.719916 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqjnj\" (UniqueName: \"kubernetes.io/projected/aaa3c20b-dc94-49b3-bb4b-3172c5f85879-kube-api-access-qqjnj\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:18:15.720128 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:15.719927 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmrw6\" (UniqueName: \"kubernetes.io/projected/8d32ebf1-5535-4c0b-bb32-dd07cdc42235-kube-api-access-fmrw6\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:18:15.817033 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:15.816995 4455 generic.go:356] "Generic (PLEG): container finished" podID="aaa3c20b-dc94-49b3-bb4b-3172c5f85879" containerID="429adae083ca4ce8fa7c2e3eb0776fb1a572d259b27b26d441af119793258a56" exitCode=0 Oct 30 18:18:15.817226 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:15.817048 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-7bc6687fcf-jpj87" Oct 30 18:18:15.817226 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:15.817089 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-7bc6687fcf-jpj87" event={"ID":"aaa3c20b-dc94-49b3-bb4b-3172c5f85879","Type":"ContainerDied","Data":"429adae083ca4ce8fa7c2e3eb0776fb1a572d259b27b26d441af119793258a56"} Oct 30 18:18:15.817226 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:15.817143 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-7bc6687fcf-jpj87" event={"ID":"aaa3c20b-dc94-49b3-bb4b-3172c5f85879","Type":"ContainerDied","Data":"6c8495e0daacc79b0e090726c2a6a022c79148382ab33af347fafdfc86010cfa"} Oct 30 18:18:15.817226 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:15.817174 4455 scope.go:122] "RemoveContainer" containerID="429adae083ca4ce8fa7c2e3eb0776fb1a572d259b27b26d441af119793258a56" Oct 30 18:18:15.818753 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:15.818711 4455 generic.go:356] "Generic (PLEG): container finished" podID="8d32ebf1-5535-4c0b-bb32-dd07cdc42235" containerID="3894e4a8c2ea219db5a30edc627f48b2a52f03d83bd2bb00db3ac5257441033a" exitCode=0 Oct 30 18:18:15.818911 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:15.818755 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-77b4f9b8b4-999hg" Oct 30 18:18:15.818911 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:15.818800 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-77b4f9b8b4-999hg" event={"ID":"8d32ebf1-5535-4c0b-bb32-dd07cdc42235","Type":"ContainerDied","Data":"3894e4a8c2ea219db5a30edc627f48b2a52f03d83bd2bb00db3ac5257441033a"} Oct 30 18:18:15.818911 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:15.818866 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-77b4f9b8b4-999hg" event={"ID":"8d32ebf1-5535-4c0b-bb32-dd07cdc42235","Type":"ContainerDied","Data":"c2da7f5df9594333d7245921789f0d4257cd90f3b386e200e8dc7ed230b6d806"} Oct 30 18:18:15.829188 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:15.829172 4455 scope.go:122] "RemoveContainer" containerID="429adae083ca4ce8fa7c2e3eb0776fb1a572d259b27b26d441af119793258a56" Oct 30 18:18:15.829498 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:18:15.829480 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"429adae083ca4ce8fa7c2e3eb0776fb1a572d259b27b26d441af119793258a56\": container with ID starting with 429adae083ca4ce8fa7c2e3eb0776fb1a572d259b27b26d441af119793258a56 not found: ID does not exist" containerID="429adae083ca4ce8fa7c2e3eb0776fb1a572d259b27b26d441af119793258a56" Oct 30 18:18:15.829564 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:15.829507 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"429adae083ca4ce8fa7c2e3eb0776fb1a572d259b27b26d441af119793258a56"} err="failed to get container status \"429adae083ca4ce8fa7c2e3eb0776fb1a572d259b27b26d441af119793258a56\": rpc error: code = NotFound desc = could not find container \"429adae083ca4ce8fa7c2e3eb0776fb1a572d259b27b26d441af119793258a56\": container with ID starting with 429adae083ca4ce8fa7c2e3eb0776fb1a572d259b27b26d441af119793258a56 not found: ID does not exist" Oct 30 18:18:15.829564 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:15.829524 4455 scope.go:122] "RemoveContainer" containerID="3894e4a8c2ea219db5a30edc627f48b2a52f03d83bd2bb00db3ac5257441033a" Oct 30 18:18:15.841489 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:15.841465 4455 scope.go:122] "RemoveContainer" containerID="3894e4a8c2ea219db5a30edc627f48b2a52f03d83bd2bb00db3ac5257441033a" Oct 30 18:18:15.841771 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:18:15.841753 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3894e4a8c2ea219db5a30edc627f48b2a52f03d83bd2bb00db3ac5257441033a\": container with ID starting with 3894e4a8c2ea219db5a30edc627f48b2a52f03d83bd2bb00db3ac5257441033a not found: ID does not exist" containerID="3894e4a8c2ea219db5a30edc627f48b2a52f03d83bd2bb00db3ac5257441033a" Oct 30 18:18:15.841835 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:15.841778 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3894e4a8c2ea219db5a30edc627f48b2a52f03d83bd2bb00db3ac5257441033a"} err="failed to get container status \"3894e4a8c2ea219db5a30edc627f48b2a52f03d83bd2bb00db3ac5257441033a\": rpc error: code = NotFound desc = could not find container \"3894e4a8c2ea219db5a30edc627f48b2a52f03d83bd2bb00db3ac5257441033a\": container with ID starting with 3894e4a8c2ea219db5a30edc627f48b2a52f03d83bd2bb00db3ac5257441033a not found: ID does not exist" Oct 30 18:18:15.878106 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:15.878071 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-853d3a-77b4f9b8b4-999hg"] Oct 30 18:18:15.890983 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:15.890958 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-853d3a-77b4f9b8b4-999hg"] Oct 30 18:18:15.918819 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:15.918782 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-7bc6687fcf-jpj87"] Oct 30 18:18:15.922667 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:15.922646 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-7bc6687fcf-jpj87"] Oct 30 18:18:17.284388 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:17.284352 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-xmhnq"] Oct 30 18:18:17.321379 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:17.321344 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-8hk9k"] Oct 30 18:18:17.638815 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:17.638741 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d32ebf1-5535-4c0b-bb32-dd07cdc42235" path="/var/lib/kubelet/pods/8d32ebf1-5535-4c0b-bb32-dd07cdc42235/volumes" Oct 30 18:18:17.639348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:17.639301 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaa3c20b-dc94-49b3-bb4b-3172c5f85879" path="/var/lib/kubelet/pods/aaa3c20b-dc94-49b3-bb4b-3172c5f85879/volumes" Oct 30 18:18:18.283007 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.282974 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v"] Oct 30 18:18:18.283858 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.283834 4455 cpu_manager.go:414] "containerMap: removing container" podUID="ebc64c0e-74bd-4130-bf28-65c0e73f2197" containerName="payload-pre-processing" Oct 30 18:18:18.283858 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.283854 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebc64c0e-74bd-4130-bf28-65c0e73f2197" containerName="payload-pre-processing" Oct 30 18:18:18.284065 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.283891 4455 cpu_manager.go:414] "containerMap: removing container" podUID="f62dc6ab-f96a-49ed-9f75-158f121781d7" containerName="payload-processing" Oct 30 18:18:18.284065 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.283899 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f62dc6ab-f96a-49ed-9f75-158f121781d7" containerName="payload-processing" Oct 30 18:18:18.284065 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.283911 4455 cpu_manager.go:414] "containerMap: removing container" podUID="8d32ebf1-5535-4c0b-bb32-dd07cdc42235" containerName="payload-processing" Oct 30 18:18:18.284065 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.283916 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d32ebf1-5535-4c0b-bb32-dd07cdc42235" containerName="payload-processing" Oct 30 18:18:18.284065 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.283934 4455 cpu_manager.go:414] "containerMap: removing container" podUID="aaa3c20b-dc94-49b3-bb4b-3172c5f85879" containerName="payload-pre-processing" Oct 30 18:18:18.284065 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.283939 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaa3c20b-dc94-49b3-bb4b-3172c5f85879" containerName="payload-pre-processing" Oct 30 18:18:18.284399 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.284089 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="f62dc6ab-f96a-49ed-9f75-158f121781d7" containerName="payload-processing" Oct 30 18:18:18.284399 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.284102 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d32ebf1-5535-4c0b-bb32-dd07cdc42235" containerName="payload-processing" Oct 30 18:18:18.284399 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.284111 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaa3c20b-dc94-49b3-bb4b-3172c5f85879" containerName="payload-pre-processing" Oct 30 18:18:18.284399 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.284122 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebc64c0e-74bd-4130-bf28-65c0e73f2197" containerName="payload-pre-processing" Oct 30 18:18:18.287877 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.287856 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v" Oct 30 18:18:18.291241 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.291211 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-6c67ff6bd-8tznm" podUID="5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5" containerName="payload-processing" containerID="cri-o://452e5d2ad27f3778e38cbefb0539323968866e2c23878a0105a5f00bed090124" gracePeriod=25 Oct 30 18:18:18.308108 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.307743 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-c33ee4-serving-cert\"" Oct 30 18:18:18.309974 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.309939 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-75cbbbf795-jcxkn" podUID="691071d4-c640-4df5-9882-c58de6741882" containerName="payload-pre-processing" containerID="cri-o://2423448c78707ec90c98efe1ea93a5c16833743e632c060190e590ab04cbe01e" gracePeriod=25 Oct 30 18:18:18.327631 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.327601 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v"] Oct 30 18:18:18.373115 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.373077 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22a3bbc2-f6ee-409d-937b-a0faf2903240-metrics-certs\") pod \"maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v\" (UID: \"22a3bbc2-f6ee-409d-937b-a0faf2903240\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v" Oct 30 18:18:18.373563 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.373540 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22a3bbc2-f6ee-409d-937b-a0faf2903240-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v\" (UID: \"22a3bbc2-f6ee-409d-937b-a0faf2903240\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v" Oct 30 18:18:18.373680 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.373661 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7fgr\" (UniqueName: \"kubernetes.io/projected/22a3bbc2-f6ee-409d-937b-a0faf2903240-kube-api-access-j7fgr\") pod \"maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v\" (UID: \"22a3bbc2-f6ee-409d-937b-a0faf2903240\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v" Oct 30 18:18:18.373747 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.373696 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a3bbc2-f6ee-409d-937b-a0faf2903240-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v\" (UID: \"22a3bbc2-f6ee-409d-937b-a0faf2903240\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v" Oct 30 18:18:18.475212 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.475164 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22a3bbc2-f6ee-409d-937b-a0faf2903240-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v\" (UID: \"22a3bbc2-f6ee-409d-937b-a0faf2903240\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v" Oct 30 18:18:18.475428 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.475238 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7fgr\" (UniqueName: \"kubernetes.io/projected/22a3bbc2-f6ee-409d-937b-a0faf2903240-kube-api-access-j7fgr\") pod \"maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v\" (UID: \"22a3bbc2-f6ee-409d-937b-a0faf2903240\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v" Oct 30 18:18:18.475428 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.475263 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a3bbc2-f6ee-409d-937b-a0faf2903240-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v\" (UID: \"22a3bbc2-f6ee-409d-937b-a0faf2903240\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v" Oct 30 18:18:18.475428 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.475299 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22a3bbc2-f6ee-409d-937b-a0faf2903240-metrics-certs\") pod \"maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v\" (UID: \"22a3bbc2-f6ee-409d-937b-a0faf2903240\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v" Oct 30 18:18:18.476998 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.476941 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a3bbc2-f6ee-409d-937b-a0faf2903240-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v\" (UID: \"22a3bbc2-f6ee-409d-937b-a0faf2903240\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v" Oct 30 18:18:18.478828 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.478802 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22a3bbc2-f6ee-409d-937b-a0faf2903240-metrics-certs\") pod \"maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v\" (UID: \"22a3bbc2-f6ee-409d-937b-a0faf2903240\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v" Oct 30 18:18:18.479876 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.479855 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22a3bbc2-f6ee-409d-937b-a0faf2903240-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v\" (UID: \"22a3bbc2-f6ee-409d-937b-a0faf2903240\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v" Oct 30 18:18:18.512377 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.512287 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7fgr\" (UniqueName: \"kubernetes.io/projected/22a3bbc2-f6ee-409d-937b-a0faf2903240-kube-api-access-j7fgr\") pod \"maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v\" (UID: \"22a3bbc2-f6ee-409d-937b-a0faf2903240\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v" Oct 30 18:18:18.728787 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.728761 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v" Oct 30 18:18:18.740348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.739279 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-6c67ff6bd-8tznm" Oct 30 18:18:18.822940 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.822909 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5-plugins-config-volume\") pod \"5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5\" (UID: \"5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5\") " Oct 30 18:18:18.823045 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.822995 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5-kube-api-access-ppbjl\" (UniqueName: \"kubernetes.io/projected/5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5-kube-api-access-ppbjl\") pod \"5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5\" (UID: \"5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5\") " Oct 30 18:18:18.826651 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.826606 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5-plugins-config-volume" pod "5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5" (UID: "5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:18.827034 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.826937 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5-kube-api-access-ppbjl" pod "5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5" (UID: "5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5"). InnerVolumeSpecName "kube-api-access-ppbjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:18.836878 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.836767 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-75cbbbf795-jcxkn" Oct 30 18:18:18.851582 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.851544 4455 generic.go:356] "Generic (PLEG): container finished" podID="691071d4-c640-4df5-9882-c58de6741882" containerID="2423448c78707ec90c98efe1ea93a5c16833743e632c060190e590ab04cbe01e" exitCode=0 Oct 30 18:18:18.851780 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.851764 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-75cbbbf795-jcxkn" Oct 30 18:18:18.851931 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.851912 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-75cbbbf795-jcxkn" event={"ID":"691071d4-c640-4df5-9882-c58de6741882","Type":"ContainerDied","Data":"2423448c78707ec90c98efe1ea93a5c16833743e632c060190e590ab04cbe01e"} Oct 30 18:18:18.851980 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.851965 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-75cbbbf795-jcxkn" event={"ID":"691071d4-c640-4df5-9882-c58de6741882","Type":"ContainerDied","Data":"0960d57cc14db8bb715122be9b973008bf2fbed6402cd9f50f30655fc7c05601"} Oct 30 18:18:18.852012 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.852002 4455 scope.go:122] "RemoveContainer" containerID="2423448c78707ec90c98efe1ea93a5c16833743e632c060190e590ab04cbe01e" Oct 30 18:18:18.854735 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.854706 4455 generic.go:356] "Generic (PLEG): container finished" podID="5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5" containerID="452e5d2ad27f3778e38cbefb0539323968866e2c23878a0105a5f00bed090124" exitCode=0 Oct 30 18:18:18.854947 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.854833 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-6c67ff6bd-8tznm" event={"ID":"5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5","Type":"ContainerDied","Data":"452e5d2ad27f3778e38cbefb0539323968866e2c23878a0105a5f00bed090124"} Oct 30 18:18:18.855125 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.855097 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-6c67ff6bd-8tznm" event={"ID":"5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5","Type":"ContainerDied","Data":"9f71eac750cf4ce755895b851691715a56c162b834e4555b33b55759838be311"} Oct 30 18:18:18.855263 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.854901 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-6c67ff6bd-8tznm" Oct 30 18:18:18.875550 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.875518 4455 scope.go:122] "RemoveContainer" containerID="2423448c78707ec90c98efe1ea93a5c16833743e632c060190e590ab04cbe01e" Oct 30 18:18:18.876129 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:18:18.875968 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2423448c78707ec90c98efe1ea93a5c16833743e632c060190e590ab04cbe01e\": container with ID starting with 2423448c78707ec90c98efe1ea93a5c16833743e632c060190e590ab04cbe01e not found: ID does not exist" containerID="2423448c78707ec90c98efe1ea93a5c16833743e632c060190e590ab04cbe01e" Oct 30 18:18:18.876129 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.876009 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2423448c78707ec90c98efe1ea93a5c16833743e632c060190e590ab04cbe01e"} err="failed to get container status \"2423448c78707ec90c98efe1ea93a5c16833743e632c060190e590ab04cbe01e\": rpc error: code = NotFound desc = could not find container \"2423448c78707ec90c98efe1ea93a5c16833743e632c060190e590ab04cbe01e\": container with ID starting with 2423448c78707ec90c98efe1ea93a5c16833743e632c060190e590ab04cbe01e not found: ID does not exist" Oct 30 18:18:18.876129 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.876033 4455 scope.go:122] "RemoveContainer" containerID="452e5d2ad27f3778e38cbefb0539323968866e2c23878a0105a5f00bed090124" Oct 30 18:18:18.899735 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.899465 4455 scope.go:122] "RemoveContainer" containerID="452e5d2ad27f3778e38cbefb0539323968866e2c23878a0105a5f00bed090124" Oct 30 18:18:18.900018 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:18:18.899990 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"452e5d2ad27f3778e38cbefb0539323968866e2c23878a0105a5f00bed090124\": container with ID starting with 452e5d2ad27f3778e38cbefb0539323968866e2c23878a0105a5f00bed090124 not found: ID does not exist" containerID="452e5d2ad27f3778e38cbefb0539323968866e2c23878a0105a5f00bed090124" Oct 30 18:18:18.900110 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.900024 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"452e5d2ad27f3778e38cbefb0539323968866e2c23878a0105a5f00bed090124"} err="failed to get container status \"452e5d2ad27f3778e38cbefb0539323968866e2c23878a0105a5f00bed090124\": rpc error: code = NotFound desc = could not find container \"452e5d2ad27f3778e38cbefb0539323968866e2c23878a0105a5f00bed090124\": container with ID starting with 452e5d2ad27f3778e38cbefb0539323968866e2c23878a0105a5f00bed090124 not found: ID does not exist" Oct 30 18:18:18.924814 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.924780 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/691071d4-c640-4df5-9882-c58de6741882-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/691071d4-c640-4df5-9882-c58de6741882-plugins-config-volume\") pod \"691071d4-c640-4df5-9882-c58de6741882\" (UID: \"691071d4-c640-4df5-9882-c58de6741882\") " Oct 30 18:18:18.924980 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.924930 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/691071d4-c640-4df5-9882-c58de6741882-kube-api-access-4s4lt\" (UniqueName: \"kubernetes.io/projected/691071d4-c640-4df5-9882-c58de6741882-kube-api-access-4s4lt\") pod \"691071d4-c640-4df5-9882-c58de6741882\" (UID: \"691071d4-c640-4df5-9882-c58de6741882\") " Oct 30 18:18:18.925873 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.925227 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/691071d4-c640-4df5-9882-c58de6741882-plugins-config-volume" pod "691071d4-c640-4df5-9882-c58de6741882" (UID: "691071d4-c640-4df5-9882-c58de6741882"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:18.925873 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.925775 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:18:18.925873 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.925796 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/691071d4-c640-4df5-9882-c58de6741882-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:18:18.925873 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.925810 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppbjl\" (UniqueName: \"kubernetes.io/projected/5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5-kube-api-access-ppbjl\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:18:18.929293 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.929266 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/691071d4-c640-4df5-9882-c58de6741882-kube-api-access-4s4lt" pod "691071d4-c640-4df5-9882-c58de6741882" (UID: "691071d4-c640-4df5-9882-c58de6741882"). InnerVolumeSpecName "kube-api-access-4s4lt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:18.930663 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.930645 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-064408-6c67ff6bd-8tznm"] Oct 30 18:18:18.935195 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:18.935172 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-064408-6c67ff6bd-8tznm"] Oct 30 18:18:19.028032 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:19.027922 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s4lt\" (UniqueName: \"kubernetes.io/projected/691071d4-c640-4df5-9882-c58de6741882-kube-api-access-4s4lt\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:18:19.062051 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:18:19.062017 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a3bbc2_f6ee_409d_937b_a0faf2903240.slice/crio-66773b0edc2884afca923a9ccb2fcb1eb2b565bb3bddef0c64b9491dd82fa539 WatchSource:0}: Error finding container 66773b0edc2884afca923a9ccb2fcb1eb2b565bb3bddef0c64b9491dd82fa539: Status 404 returned error can't find the container with id 66773b0edc2884afca923a9ccb2fcb1eb2b565bb3bddef0c64b9491dd82fa539 Oct 30 18:18:19.063572 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:19.063550 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v"] Oct 30 18:18:19.254362 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:19.254285 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-75cbbbf795-jcxkn"] Oct 30 18:18:19.258208 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:19.258178 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-064408-75cbbbf795-jcxkn"] Oct 30 18:18:19.707398 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:19.704554 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5" path="/var/lib/kubelet/pods/5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5/volumes" Oct 30 18:18:19.707398 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:19.705474 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="691071d4-c640-4df5-9882-c58de6741882" path="/var/lib/kubelet/pods/691071d4-c640-4df5-9882-c58de6741882/volumes" Oct 30 18:18:19.868963 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:19.868927 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v" event={"ID":"22a3bbc2-f6ee-409d-937b-a0faf2903240","Type":"ContainerStarted","Data":"e11b301b3129ae62b37873f32f59d745e2faed3caac9a1cc190f424d54db2ca8"} Oct 30 18:18:19.869183 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:19.868997 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v" event={"ID":"22a3bbc2-f6ee-409d-937b-a0faf2903240","Type":"ContainerStarted","Data":"66773b0edc2884afca923a9ccb2fcb1eb2b565bb3bddef0c64b9491dd82fa539"} Oct 30 18:18:19.869183 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:19.869113 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v" Oct 30 18:18:19.916190 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:19.916135 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v" podStartSLOduration=1.5761138670000001 podStartE2EDuration="1.916112577s" totalImagesPullingTime="339.99871ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:19.914399117 +0000 UTC m=+1552.402036152" watchObservedRunningTime="2026-10-30 18:18:19.916112577 +0000 UTC m=+1552.403749611" Oct 30 18:18:22.286347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:22.285019 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-xmhnq" podUID="72fd9184-e60c-4e6d-8415-e6e146e8b5a2" containerName="payload-processing" containerID="cri-o://3b121fdd61d3d5d5aa2e6d9477b94911d97a9644a1426897d9efa03c396d2929" gracePeriod=25 Oct 30 18:18:22.322553 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:22.322510 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-8hk9k" podUID="3357893d-bb50-423d-95f8-2b7c3bd7dc5e" containerName="payload-pre-processing" containerID="cri-o://927022c133242a12b425b739630114283a31396c1abdefce516c0bf7e44036cd" gracePeriod=25 Oct 30 18:18:22.620872 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:22.620839 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-xmhnq" Oct 30 18:18:22.685981 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:22.685956 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72fd9184-e60c-4e6d-8415-e6e146e8b5a2-kube-api-access-rsxgw\" (UniqueName: \"kubernetes.io/projected/72fd9184-e60c-4e6d-8415-e6e146e8b5a2-kube-api-access-rsxgw\") pod \"72fd9184-e60c-4e6d-8415-e6e146e8b5a2\" (UID: \"72fd9184-e60c-4e6d-8415-e6e146e8b5a2\") " Oct 30 18:18:22.686145 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:22.686123 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72fd9184-e60c-4e6d-8415-e6e146e8b5a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72fd9184-e60c-4e6d-8415-e6e146e8b5a2-plugins-config-volume\") pod \"72fd9184-e60c-4e6d-8415-e6e146e8b5a2\" (UID: \"72fd9184-e60c-4e6d-8415-e6e146e8b5a2\") " Oct 30 18:18:22.686673 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:22.686649 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72fd9184-e60c-4e6d-8415-e6e146e8b5a2-plugins-config-volume" pod "72fd9184-e60c-4e6d-8415-e6e146e8b5a2" (UID: "72fd9184-e60c-4e6d-8415-e6e146e8b5a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:22.689221 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:22.689199 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72fd9184-e60c-4e6d-8415-e6e146e8b5a2-kube-api-access-rsxgw" pod "72fd9184-e60c-4e6d-8415-e6e146e8b5a2" (UID: "72fd9184-e60c-4e6d-8415-e6e146e8b5a2"). InnerVolumeSpecName "kube-api-access-rsxgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:22.716362 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:22.712479 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-8hk9k" Oct 30 18:18:22.788147 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:22.788111 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3357893d-bb50-423d-95f8-2b7c3bd7dc5e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3357893d-bb50-423d-95f8-2b7c3bd7dc5e-plugins-config-volume\") pod \"3357893d-bb50-423d-95f8-2b7c3bd7dc5e\" (UID: \"3357893d-bb50-423d-95f8-2b7c3bd7dc5e\") " Oct 30 18:18:22.788339 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:22.788218 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3357893d-bb50-423d-95f8-2b7c3bd7dc5e-kube-api-access-pvn2h\" (UniqueName: \"kubernetes.io/projected/3357893d-bb50-423d-95f8-2b7c3bd7dc5e-kube-api-access-pvn2h\") pod \"3357893d-bb50-423d-95f8-2b7c3bd7dc5e\" (UID: \"3357893d-bb50-423d-95f8-2b7c3bd7dc5e\") " Oct 30 18:18:22.788621 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:22.788598 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3357893d-bb50-423d-95f8-2b7c3bd7dc5e-plugins-config-volume" pod "3357893d-bb50-423d-95f8-2b7c3bd7dc5e" (UID: "3357893d-bb50-423d-95f8-2b7c3bd7dc5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:22.788872 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:22.788851 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3357893d-bb50-423d-95f8-2b7c3bd7dc5e-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:18:22.788872 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:22.788872 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72fd9184-e60c-4e6d-8415-e6e146e8b5a2-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:18:22.789030 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:22.788887 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsxgw\" (UniqueName: \"kubernetes.io/projected/72fd9184-e60c-4e6d-8415-e6e146e8b5a2-kube-api-access-rsxgw\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:18:22.790658 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:22.790629 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3357893d-bb50-423d-95f8-2b7c3bd7dc5e-kube-api-access-pvn2h" pod "3357893d-bb50-423d-95f8-2b7c3bd7dc5e" (UID: "3357893d-bb50-423d-95f8-2b7c3bd7dc5e"). InnerVolumeSpecName "kube-api-access-pvn2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:22.890065 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:22.889980 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvn2h\" (UniqueName: \"kubernetes.io/projected/3357893d-bb50-423d-95f8-2b7c3bd7dc5e-kube-api-access-pvn2h\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:18:22.895230 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:22.895205 4455 generic.go:356] "Generic (PLEG): container finished" podID="72fd9184-e60c-4e6d-8415-e6e146e8b5a2" containerID="3b121fdd61d3d5d5aa2e6d9477b94911d97a9644a1426897d9efa03c396d2929" exitCode=0 Oct 30 18:18:22.895389 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:22.895259 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-xmhnq" Oct 30 18:18:22.895389 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:22.895292 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-xmhnq" event={"ID":"72fd9184-e60c-4e6d-8415-e6e146e8b5a2","Type":"ContainerDied","Data":"3b121fdd61d3d5d5aa2e6d9477b94911d97a9644a1426897d9efa03c396d2929"} Oct 30 18:18:22.895389 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:22.895385 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-xmhnq" event={"ID":"72fd9184-e60c-4e6d-8415-e6e146e8b5a2","Type":"ContainerDied","Data":"54685b6ce40e4d6c33cc2f46ef2f3f87a3d71e64676b80faf5482e80a426c2d8"} Oct 30 18:18:22.895538 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:22.895414 4455 scope.go:122] "RemoveContainer" containerID="3b121fdd61d3d5d5aa2e6d9477b94911d97a9644a1426897d9efa03c396d2929" Oct 30 18:18:22.896825 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:22.896801 4455 generic.go:356] "Generic (PLEG): container finished" podID="3357893d-bb50-423d-95f8-2b7c3bd7dc5e" containerID="927022c133242a12b425b739630114283a31396c1abdefce516c0bf7e44036cd" exitCode=0 Oct 30 18:18:22.896944 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:22.896843 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-8hk9k" Oct 30 18:18:22.896944 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:22.896881 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-8hk9k" event={"ID":"3357893d-bb50-423d-95f8-2b7c3bd7dc5e","Type":"ContainerDied","Data":"927022c133242a12b425b739630114283a31396c1abdefce516c0bf7e44036cd"} Oct 30 18:18:22.896944 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:22.896928 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-8hk9k" event={"ID":"3357893d-bb50-423d-95f8-2b7c3bd7dc5e","Type":"ContainerDied","Data":"4e8880f0cba7b92647d2def17a1489ab750693322006b783811a277efa6654a9"} Oct 30 18:18:22.907894 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:22.907863 4455 scope.go:122] "RemoveContainer" containerID="3b121fdd61d3d5d5aa2e6d9477b94911d97a9644a1426897d9efa03c396d2929" Oct 30 18:18:22.908155 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:18:22.908139 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b121fdd61d3d5d5aa2e6d9477b94911d97a9644a1426897d9efa03c396d2929\": container with ID starting with 3b121fdd61d3d5d5aa2e6d9477b94911d97a9644a1426897d9efa03c396d2929 not found: ID does not exist" containerID="3b121fdd61d3d5d5aa2e6d9477b94911d97a9644a1426897d9efa03c396d2929" Oct 30 18:18:22.908214 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:22.908161 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b121fdd61d3d5d5aa2e6d9477b94911d97a9644a1426897d9efa03c396d2929"} err="failed to get container status \"3b121fdd61d3d5d5aa2e6d9477b94911d97a9644a1426897d9efa03c396d2929\": rpc error: code = NotFound desc = could not find container \"3b121fdd61d3d5d5aa2e6d9477b94911d97a9644a1426897d9efa03c396d2929\": container with ID starting with 3b121fdd61d3d5d5aa2e6d9477b94911d97a9644a1426897d9efa03c396d2929 not found: ID does not exist" Oct 30 18:18:22.908214 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:22.908176 4455 scope.go:122] "RemoveContainer" containerID="927022c133242a12b425b739630114283a31396c1abdefce516c0bf7e44036cd" Oct 30 18:18:22.917840 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:22.917821 4455 scope.go:122] "RemoveContainer" containerID="927022c133242a12b425b739630114283a31396c1abdefce516c0bf7e44036cd" Oct 30 18:18:22.918070 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:18:22.918050 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"927022c133242a12b425b739630114283a31396c1abdefce516c0bf7e44036cd\": container with ID starting with 927022c133242a12b425b739630114283a31396c1abdefce516c0bf7e44036cd not found: ID does not exist" containerID="927022c133242a12b425b739630114283a31396c1abdefce516c0bf7e44036cd" Oct 30 18:18:22.918114 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:22.918076 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"927022c133242a12b425b739630114283a31396c1abdefce516c0bf7e44036cd"} err="failed to get container status \"927022c133242a12b425b739630114283a31396c1abdefce516c0bf7e44036cd\": rpc error: code = NotFound desc = could not find container \"927022c133242a12b425b739630114283a31396c1abdefce516c0bf7e44036cd\": container with ID starting with 927022c133242a12b425b739630114283a31396c1abdefce516c0bf7e44036cd not found: ID does not exist" Oct 30 18:18:22.956474 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:22.956431 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-xmhnq"] Oct 30 18:18:22.961491 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:22.961468 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c6b2120f-5666cb5864-xmhnq"] Oct 30 18:18:22.999220 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:22.999192 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-8hk9k"] Oct 30 18:18:23.002972 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:23.002948 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c6b2120f-55d49bcd99-8hk9k"] Oct 30 18:18:23.635775 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:23.635701 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3357893d-bb50-423d-95f8-2b7c3bd7dc5e" path="/var/lib/kubelet/pods/3357893d-bb50-423d-95f8-2b7c3bd7dc5e/volumes" Oct 30 18:18:23.636158 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:23.636100 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72fd9184-e60c-4e6d-8415-e6e146e8b5a2" path="/var/lib/kubelet/pods/72fd9184-e60c-4e6d-8415-e6e146e8b5a2/volumes" Oct 30 18:18:25.219265 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:25.219228 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-d5df8bc9-k4tss"] Oct 30 18:18:25.222474 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:25.220944 4455 cpu_manager.go:414] "containerMap: removing container" podUID="691071d4-c640-4df5-9882-c58de6741882" containerName="payload-pre-processing" Oct 30 18:18:25.222757 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:25.222721 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="691071d4-c640-4df5-9882-c58de6741882" containerName="payload-pre-processing" Oct 30 18:18:25.222918 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:25.222905 4455 cpu_manager.go:414] "containerMap: removing container" podUID="3357893d-bb50-423d-95f8-2b7c3bd7dc5e" containerName="payload-pre-processing" Oct 30 18:18:25.223008 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:25.222996 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3357893d-bb50-423d-95f8-2b7c3bd7dc5e" containerName="payload-pre-processing" Oct 30 18:18:25.223163 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:25.223151 4455 cpu_manager.go:414] "containerMap: removing container" podUID="72fd9184-e60c-4e6d-8415-e6e146e8b5a2" containerName="payload-processing" Oct 30 18:18:25.223246 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:25.223235 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72fd9184-e60c-4e6d-8415-e6e146e8b5a2" containerName="payload-processing" Oct 30 18:18:25.223354 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:25.223342 4455 cpu_manager.go:414] "containerMap: removing container" podUID="5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5" containerName="payload-processing" Oct 30 18:18:25.223443 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:25.223432 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5" containerName="payload-processing" Oct 30 18:18:25.223709 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:25.223695 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="691071d4-c640-4df5-9882-c58de6741882" containerName="payload-pre-processing" Oct 30 18:18:25.223799 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:25.223789 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="3357893d-bb50-423d-95f8-2b7c3bd7dc5e" containerName="payload-pre-processing" Oct 30 18:18:25.223889 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:25.223879 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="72fd9184-e60c-4e6d-8415-e6e146e8b5a2" containerName="payload-processing" Oct 30 18:18:25.223965 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:25.223955 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ddda28e-f4e1-4e1b-b229-ebb2b1dd15a5" containerName="payload-processing" Oct 30 18:18:25.232335 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:25.232285 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-d5df8bc9-k4tss" Oct 30 18:18:25.257094 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:25.257043 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-853d3a\"" Oct 30 18:18:25.263049 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:25.259092 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-853d3a-dockercfg-jpgqp\"" Oct 30 18:18:25.263049 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:25.262788 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-d5df8bc9-k4tss"] Oct 30 18:18:25.330348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:25.329479 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b16e7e76-d4fc-4844-9c0b-767c2ceed45e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-853d3a-d5df8bc9-k4tss\" (UID: \"b16e7e76-d4fc-4844-9c0b-767c2ceed45e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-d5df8bc9-k4tss" Oct 30 18:18:25.330348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:25.329605 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvvp8\" (UniqueName: \"kubernetes.io/projected/b16e7e76-d4fc-4844-9c0b-767c2ceed45e-kube-api-access-tvvp8\") pod \"payload-pre-processing-e2e-ipp-a-853d3a-d5df8bc9-k4tss\" (UID: \"b16e7e76-d4fc-4844-9c0b-767c2ceed45e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-d5df8bc9-k4tss" Oct 30 18:18:25.472751 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:25.472657 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b16e7e76-d4fc-4844-9c0b-767c2ceed45e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-853d3a-d5df8bc9-k4tss\" (UID: \"b16e7e76-d4fc-4844-9c0b-767c2ceed45e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-d5df8bc9-k4tss" Oct 30 18:18:25.473047 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:25.473027 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvvp8\" (UniqueName: \"kubernetes.io/projected/b16e7e76-d4fc-4844-9c0b-767c2ceed45e-kube-api-access-tvvp8\") pod \"payload-pre-processing-e2e-ipp-a-853d3a-d5df8bc9-k4tss\" (UID: \"b16e7e76-d4fc-4844-9c0b-767c2ceed45e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-d5df8bc9-k4tss" Oct 30 18:18:25.477743 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:25.477644 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b16e7e76-d4fc-4844-9c0b-767c2ceed45e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-853d3a-d5df8bc9-k4tss\" (UID: \"b16e7e76-d4fc-4844-9c0b-767c2ceed45e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-d5df8bc9-k4tss" Oct 30 18:18:25.501082 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:25.501053 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-853d3a-5dbcc7fb74-w5g67"] Oct 30 18:18:25.555405 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:25.547517 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvvp8\" (UniqueName: \"kubernetes.io/projected/b16e7e76-d4fc-4844-9c0b-767c2ceed45e-kube-api-access-tvvp8\") pod \"payload-pre-processing-e2e-ipp-a-853d3a-d5df8bc9-k4tss\" (UID: \"b16e7e76-d4fc-4844-9c0b-767c2ceed45e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-d5df8bc9-k4tss" Oct 30 18:18:25.562382 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:25.548964 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-853d3a-5dbcc7fb74-w5g67"] Oct 30 18:18:25.562382 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:25.549051 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-5dbcc7fb74-w5g67" Oct 30 18:18:25.580199 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:25.578708 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6lht\" (UniqueName: \"kubernetes.io/projected/8e262b82-061e-4084-a5e9-b133fdfe01da-kube-api-access-c6lht\") pod \"payload-processing-e2e-ipp-a-853d3a-5dbcc7fb74-w5g67\" (UID: \"8e262b82-061e-4084-a5e9-b133fdfe01da\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-5dbcc7fb74-w5g67" Oct 30 18:18:25.580199 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:25.578780 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e262b82-061e-4084-a5e9-b133fdfe01da-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-853d3a-5dbcc7fb74-w5g67\" (UID: \"8e262b82-061e-4084-a5e9-b133fdfe01da\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-5dbcc7fb74-w5g67" Oct 30 18:18:25.680261 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:25.680223 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6lht\" (UniqueName: \"kubernetes.io/projected/8e262b82-061e-4084-a5e9-b133fdfe01da-kube-api-access-c6lht\") pod \"payload-processing-e2e-ipp-a-853d3a-5dbcc7fb74-w5g67\" (UID: \"8e262b82-061e-4084-a5e9-b133fdfe01da\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-5dbcc7fb74-w5g67" Oct 30 18:18:25.680510 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:25.680284 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e262b82-061e-4084-a5e9-b133fdfe01da-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-853d3a-5dbcc7fb74-w5g67\" (UID: \"8e262b82-061e-4084-a5e9-b133fdfe01da\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-5dbcc7fb74-w5g67" Oct 30 18:18:25.681169 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:25.681144 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e262b82-061e-4084-a5e9-b133fdfe01da-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-853d3a-5dbcc7fb74-w5g67\" (UID: \"8e262b82-061e-4084-a5e9-b133fdfe01da\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-5dbcc7fb74-w5g67" Oct 30 18:18:25.706011 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:25.705976 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6lht\" (UniqueName: \"kubernetes.io/projected/8e262b82-061e-4084-a5e9-b133fdfe01da-kube-api-access-c6lht\") pod \"payload-processing-e2e-ipp-a-853d3a-5dbcc7fb74-w5g67\" (UID: \"8e262b82-061e-4084-a5e9-b133fdfe01da\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-5dbcc7fb74-w5g67" Oct 30 18:18:25.799878 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:25.799777 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-d5df8bc9-k4tss" Oct 30 18:18:25.886557 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:25.885949 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v" Oct 30 18:18:25.899139 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:25.898799 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-5dbcc7fb74-w5g67" Oct 30 18:18:26.164394 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:18:26.164366 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb16e7e76_d4fc_4844_9c0b_767c2ceed45e.slice/crio-25b8beccaaaaf85dd125307938d8c528958a2b2efae15f159aff7aa47529d299 WatchSource:0}: Error finding container 25b8beccaaaaf85dd125307938d8c528958a2b2efae15f159aff7aa47529d299: Status 404 returned error can't find the container with id 25b8beccaaaaf85dd125307938d8c528958a2b2efae15f159aff7aa47529d299 Oct 30 18:18:26.169191 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:26.169164 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-d5df8bc9-k4tss"] Oct 30 18:18:26.354454 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:18:26.354423 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e262b82_061e_4084_a5e9_b133fdfe01da.slice/crio-d68ea5147d17a0e3a19b6f817bf07d52d3c73ae215694f476fa2edfa7577385c WatchSource:0}: Error finding container d68ea5147d17a0e3a19b6f817bf07d52d3c73ae215694f476fa2edfa7577385c: Status 404 returned error can't find the container with id d68ea5147d17a0e3a19b6f817bf07d52d3c73ae215694f476fa2edfa7577385c Oct 30 18:18:26.355360 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:26.355072 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-853d3a-5dbcc7fb74-w5g67"] Oct 30 18:18:27.152030 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:27.151984 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-5dbcc7fb74-w5g67" event={"ID":"8e262b82-061e-4084-a5e9-b133fdfe01da","Type":"ContainerStarted","Data":"d68ea5147d17a0e3a19b6f817bf07d52d3c73ae215694f476fa2edfa7577385c"} Oct 30 18:18:27.153697 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:27.153666 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-d5df8bc9-k4tss" event={"ID":"b16e7e76-d4fc-4844-9c0b-767c2ceed45e","Type":"ContainerStarted","Data":"25b8beccaaaaf85dd125307938d8c528958a2b2efae15f159aff7aa47529d299"} Oct 30 18:18:28.163066 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:28.162968 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-5dbcc7fb74-w5g67" event={"ID":"8e262b82-061e-4084-a5e9-b133fdfe01da","Type":"ContainerStarted","Data":"5d0d6720550512af9898a55a41880fad8278aa2c1373c5f25164c7b953224ae2"} Oct 30 18:18:28.163675 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:28.163094 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-5dbcc7fb74-w5g67" Oct 30 18:18:28.164877 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:28.164850 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-d5df8bc9-k4tss" event={"ID":"b16e7e76-d4fc-4844-9c0b-767c2ceed45e","Type":"ContainerStarted","Data":"d6b50ea9c64c134569c709726081cef1624d335cf34f5e9d54fdcd890f9a1af2"} Oct 30 18:18:28.165002 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:28.164989 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-d5df8bc9-k4tss" Oct 30 18:18:28.211160 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:28.211112 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-5dbcc7fb74-w5g67" podStartSLOduration=1.712794901 podStartE2EDuration="3.211089918s" totalImagesPullingTime="1.498295017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:28.210544817 +0000 UTC m=+1560.698181853" watchObservedRunningTime="2026-10-30 18:18:28.211089918 +0000 UTC m=+1560.698726952" Oct 30 18:18:28.257598 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:28.257534 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-d5df8bc9-k4tss" podStartSLOduration=1.397824362 podStartE2EDuration="3.257517672s" totalImagesPullingTime="1.85969331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:28.254567452 +0000 UTC m=+1560.742204489" watchObservedRunningTime="2026-10-30 18:18:28.257517672 +0000 UTC m=+1560.745154704" Oct 30 18:18:29.328407 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:29.325158 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-064408-c74cbb9f4-sct25"] Oct 30 18:18:29.331590 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:29.331567 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-c74cbb9f4-sct25" Oct 30 18:18:29.343810 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:29.343779 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-064408-c74cbb9f4-sct25"] Oct 30 18:18:29.365292 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:29.365261 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-064408\"" Oct 30 18:18:29.365428 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:29.365373 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-064408-dockercfg-bzqzh\"" Oct 30 18:18:29.406939 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:29.406912 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3383bd2-d483-4239-8b12-879f8508d63a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-064408-c74cbb9f4-sct25\" (UID: \"c3383bd2-d483-4239-8b12-879f8508d63a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-c74cbb9f4-sct25" Oct 30 18:18:29.407307 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:29.407291 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klh6g\" (UniqueName: \"kubernetes.io/projected/c3383bd2-d483-4239-8b12-879f8508d63a-kube-api-access-klh6g\") pod \"payload-processing-e2e-ipp-a-064408-c74cbb9f4-sct25\" (UID: \"c3383bd2-d483-4239-8b12-879f8508d63a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-c74cbb9f4-sct25" Oct 30 18:18:29.509115 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:29.509082 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3383bd2-d483-4239-8b12-879f8508d63a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-064408-c74cbb9f4-sct25\" (UID: \"c3383bd2-d483-4239-8b12-879f8508d63a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-c74cbb9f4-sct25" Oct 30 18:18:29.509337 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:29.509134 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klh6g\" (UniqueName: \"kubernetes.io/projected/c3383bd2-d483-4239-8b12-879f8508d63a-kube-api-access-klh6g\") pod \"payload-processing-e2e-ipp-a-064408-c74cbb9f4-sct25\" (UID: \"c3383bd2-d483-4239-8b12-879f8508d63a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-c74cbb9f4-sct25" Oct 30 18:18:29.509940 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:29.509920 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3383bd2-d483-4239-8b12-879f8508d63a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-064408-c74cbb9f4-sct25\" (UID: \"c3383bd2-d483-4239-8b12-879f8508d63a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-c74cbb9f4-sct25" Oct 30 18:18:29.534920 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:29.534890 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klh6g\" (UniqueName: \"kubernetes.io/projected/c3383bd2-d483-4239-8b12-879f8508d63a-kube-api-access-klh6g\") pod \"payload-processing-e2e-ipp-a-064408-c74cbb9f4-sct25\" (UID: \"c3383bd2-d483-4239-8b12-879f8508d63a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-c74cbb9f4-sct25" Oct 30 18:18:29.661188 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:29.661116 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-c74cbb9f4-sct25" Oct 30 18:18:29.993483 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:29.993361 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-064408-c74cbb9f4-sct25"] Oct 30 18:18:30.349193 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:30.349107 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-c74cbb9f4-sct25" event={"ID":"c3383bd2-d483-4239-8b12-879f8508d63a","Type":"ContainerStarted","Data":"776a79f202bd45fad1c6c9aa7100a1a2ca71518d8471451c86047b86693d9fdc"} Oct 30 18:18:30.349193 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:30.349167 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-c74cbb9f4-sct25" event={"ID":"c3383bd2-d483-4239-8b12-879f8508d63a","Type":"ContainerStarted","Data":"24089a2bc31cd65a047972c0afb9baae0cf575a77d248fe42fb195cd9ab01aca"} Oct 30 18:18:30.349702 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:30.349336 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-c74cbb9f4-sct25" Oct 30 18:18:30.394761 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:18:30.394717 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:18:30.396125 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:30.396051 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-c74cbb9f4-sct25" podStartSLOduration=1.396033629 podStartE2EDuration="1.396033629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:30.393384068 +0000 UTC m=+1562.881021103" watchObservedRunningTime="2026-10-30 18:18:30.396033629 +0000 UTC m=+1562.883670665" Oct 30 18:18:33.065368 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:33.059887 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9"] Oct 30 18:18:33.065368 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:33.060205 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9" podUID="fd1096fc-1f87-4e00-b6ec-2195cb6be82a" containerName="maas-api" containerID="cri-o://2d392a5e47fb00522c705813e1e71ae6c9bc151d03e82e991b45324ca3df6f77" gracePeriod=30 Oct 30 18:18:34.771351 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:34.767504 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9" Oct 30 18:18:34.780441 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:34.776967 4455 generic.go:356] "Generic (PLEG): container finished" podID="fd1096fc-1f87-4e00-b6ec-2195cb6be82a" containerID="2d392a5e47fb00522c705813e1e71ae6c9bc151d03e82e991b45324ca3df6f77" exitCode=0 Oct 30 18:18:34.780441 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:34.777179 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9" event={"ID":"fd1096fc-1f87-4e00-b6ec-2195cb6be82a","Type":"ContainerDied","Data":"2d392a5e47fb00522c705813e1e71ae6c9bc151d03e82e991b45324ca3df6f77"} Oct 30 18:18:34.780441 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:34.777233 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9" event={"ID":"fd1096fc-1f87-4e00-b6ec-2195cb6be82a","Type":"ContainerDied","Data":"a3766fc14ef49903c4866d5bc4bbf248393a74173c8768851096306ebd0b177e"} Oct 30 18:18:34.780441 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:34.777264 4455 scope.go:122] "RemoveContainer" containerID="2d392a5e47fb00522c705813e1e71ae6c9bc151d03e82e991b45324ca3df6f77" Oct 30 18:18:34.780441 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:34.777474 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9" Oct 30 18:18:34.834344 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:34.834284 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd1096fc-1f87-4e00-b6ec-2195cb6be82a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1096fc-1f87-4e00-b6ec-2195cb6be82a-openshift-service-ca\") pod \"fd1096fc-1f87-4e00-b6ec-2195cb6be82a\" (UID: \"fd1096fc-1f87-4e00-b6ec-2195cb6be82a\") " Oct 30 18:18:34.834670 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:34.834638 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd1096fc-1f87-4e00-b6ec-2195cb6be82a-kube-api-access-tkz97\" (UniqueName: \"kubernetes.io/projected/fd1096fc-1f87-4e00-b6ec-2195cb6be82a-kube-api-access-tkz97\") pod \"fd1096fc-1f87-4e00-b6ec-2195cb6be82a\" (UID: \"fd1096fc-1f87-4e00-b6ec-2195cb6be82a\") " Oct 30 18:18:34.834849 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:34.834831 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd1096fc-1f87-4e00-b6ec-2195cb6be82a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd1096fc-1f87-4e00-b6ec-2195cb6be82a-maas-api-tls\") pod \"fd1096fc-1f87-4e00-b6ec-2195cb6be82a\" (UID: \"fd1096fc-1f87-4e00-b6ec-2195cb6be82a\") " Oct 30 18:18:34.838383 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:34.838353 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd1096fc-1f87-4e00-b6ec-2195cb6be82a-openshift-service-ca" pod "fd1096fc-1f87-4e00-b6ec-2195cb6be82a" (UID: "fd1096fc-1f87-4e00-b6ec-2195cb6be82a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:34.865346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:34.862557 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd1096fc-1f87-4e00-b6ec-2195cb6be82a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd1096fc-1f87-4e00-b6ec-2195cb6be82a-metrics-certs\") pod \"fd1096fc-1f87-4e00-b6ec-2195cb6be82a\" (UID: \"fd1096fc-1f87-4e00-b6ec-2195cb6be82a\") " Oct 30 18:18:34.865346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:34.863817 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1096fc-1f87-4e00-b6ec-2195cb6be82a-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:18:34.906362 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:34.879853 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd1096fc-1f87-4e00-b6ec-2195cb6be82a-kube-api-access-tkz97" pod "fd1096fc-1f87-4e00-b6ec-2195cb6be82a" (UID: "fd1096fc-1f87-4e00-b6ec-2195cb6be82a"). InnerVolumeSpecName "kube-api-access-tkz97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:35.223434 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:35.203352 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkz97\" (UniqueName: \"kubernetes.io/projected/fd1096fc-1f87-4e00-b6ec-2195cb6be82a-kube-api-access-tkz97\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:18:35.227344 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:35.224659 4455 scope.go:122] "RemoveContainer" containerID="2d392a5e47fb00522c705813e1e71ae6c9bc151d03e82e991b45324ca3df6f77" Oct 30 18:18:35.273346 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:18:35.269572 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d392a5e47fb00522c705813e1e71ae6c9bc151d03e82e991b45324ca3df6f77\": container with ID starting with 2d392a5e47fb00522c705813e1e71ae6c9bc151d03e82e991b45324ca3df6f77 not found: ID does not exist" containerID="2d392a5e47fb00522c705813e1e71ae6c9bc151d03e82e991b45324ca3df6f77" Oct 30 18:18:35.273346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:35.269638 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d392a5e47fb00522c705813e1e71ae6c9bc151d03e82e991b45324ca3df6f77"} err="failed to get container status \"2d392a5e47fb00522c705813e1e71ae6c9bc151d03e82e991b45324ca3df6f77\": rpc error: code = NotFound desc = could not find container \"2d392a5e47fb00522c705813e1e71ae6c9bc151d03e82e991b45324ca3df6f77\": container with ID starting with 2d392a5e47fb00522c705813e1e71ae6c9bc151d03e82e991b45324ca3df6f77 not found: ID does not exist" Oct 30 18:18:35.287558 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:35.287518 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd1096fc-1f87-4e00-b6ec-2195cb6be82a-maas-api-tls" pod "fd1096fc-1f87-4e00-b6ec-2195cb6be82a" (UID: "fd1096fc-1f87-4e00-b6ec-2195cb6be82a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:35.351350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:35.318046 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd1096fc-1f87-4e00-b6ec-2195cb6be82a-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:18:35.401592 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:35.401537 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd1096fc-1f87-4e00-b6ec-2195cb6be82a-metrics-certs" pod "fd1096fc-1f87-4e00-b6ec-2195cb6be82a" (UID: "fd1096fc-1f87-4e00-b6ec-2195cb6be82a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:35.503860 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:35.503800 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd1096fc-1f87-4e00-b6ec-2195cb6be82a-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:18:36.356770 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:36.356714 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:18:37.428379 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:37.425866 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:18:37.469830 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:37.463835 4455 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:37.490521 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:18:37.490485 4455 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.818s" Oct 30 18:18:37.507536 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:37.507503 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9"] Oct 30 18:18:37.511349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:37.507815 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:18:38.186474 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:38.177280 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c6b2120f-5bd7d6d5dc-zlds9"] Oct 30 18:18:38.432606 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:38.432470 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:38.757416 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:18:38.757316 4455 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e262b82_061e_4084_a5e9_b133fdfe01da.slice/crio-conmon-5d0d6720550512af9898a55a41880fad8278aa2c1373c5f25164c7b953224ae2.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e262b82_061e_4084_a5e9_b133fdfe01da.slice/crio-5d0d6720550512af9898a55a41880fad8278aa2c1373c5f25164c7b953224ae2.scope\": RecentStats: unable to find data in memory cache]" Oct 30 18:18:39.018914 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:39.018828 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:18:39.167384 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:39.167348 4455 generic.go:356] "Generic (PLEG): container finished" podID="8e262b82-061e-4084-a5e9-b133fdfe01da" containerID="5d0d6720550512af9898a55a41880fad8278aa2c1373c5f25164c7b953224ae2" exitCode=1 Oct 30 18:18:39.168142 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:39.168115 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-5dbcc7fb74-w5g67" event={"ID":"8e262b82-061e-4084-a5e9-b133fdfe01da","Type":"ContainerDied","Data":"5d0d6720550512af9898a55a41880fad8278aa2c1373c5f25164c7b953224ae2"} Oct 30 18:18:39.170313 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:39.170288 4455 scope.go:122] "RemoveContainer" containerID="5d0d6720550512af9898a55a41880fad8278aa2c1373c5f25164c7b953224ae2" Oct 30 18:18:39.347059 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:39.346943 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-d5df8bc9-k4tss" Oct 30 18:18:39.867481 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:39.867442 4455 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:18:39.969383 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:39.969295 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" podUID="d2c2d808-5274-484e-8520-1f88fa79fc47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:40.241648 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:40.227282 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd1096fc-1f87-4e00-b6ec-2195cb6be82a" path="/var/lib/kubelet/pods/fd1096fc-1f87-4e00-b6ec-2195cb6be82a/volumes" Oct 30 18:18:40.475347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:40.449961 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:41.213390 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:41.213354 4455 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:18:41.236436 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:41.235567 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" podUID="b933cb99-2e87-4707-a33b-de09b5c03536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:41.365508 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:41.365470 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-c74cbb9f4-sct25" Oct 30 18:18:42.123349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:42.101495 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-5dbcc7fb74-w5g67" event={"ID":"8e262b82-061e-4084-a5e9-b133fdfe01da","Type":"ContainerStarted","Data":"f95b640388a89f7f12a8bb90412fa45e57c429e35fde3d506895862cfe68f190"} Oct 30 18:18:42.124354 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:42.124303 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-5dbcc7fb74-w5g67" Oct 30 18:18:53.219383 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:53.219312 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-5dbcc7fb74-w5g67" Oct 30 18:18:57.208657 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:57.208606 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp"] Oct 30 18:18:57.209928 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:57.209901 4455 cpu_manager.go:414] "containerMap: removing container" podUID="fd1096fc-1f87-4e00-b6ec-2195cb6be82a" containerName="maas-api" Oct 30 18:18:57.209928 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:57.209927 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd1096fc-1f87-4e00-b6ec-2195cb6be82a" containerName="maas-api" Oct 30 18:18:57.210164 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:57.210101 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd1096fc-1f87-4e00-b6ec-2195cb6be82a" containerName="maas-api" Oct 30 18:18:57.218371 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:57.218314 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp" Oct 30 18:18:57.229364 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:57.229160 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp"] Oct 30 18:18:57.237357 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:57.237307 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-d951c4-serving-cert\"" Oct 30 18:18:57.249526 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:57.249495 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js8zc\" (UniqueName: \"kubernetes.io/projected/dd45faeb-139e-4756-9b91-4ed08d919e39-kube-api-access-js8zc\") pod \"maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp\" (UID: \"dd45faeb-139e-4756-9b91-4ed08d919e39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp" Oct 30 18:18:57.249850 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:57.249827 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd45faeb-139e-4756-9b91-4ed08d919e39-metrics-certs\") pod \"maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp\" (UID: \"dd45faeb-139e-4756-9b91-4ed08d919e39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp" Oct 30 18:18:57.249931 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:57.249884 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd45faeb-139e-4756-9b91-4ed08d919e39-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp\" (UID: \"dd45faeb-139e-4756-9b91-4ed08d919e39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp" Oct 30 18:18:57.250044 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:57.250017 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd45faeb-139e-4756-9b91-4ed08d919e39-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp\" (UID: \"dd45faeb-139e-4756-9b91-4ed08d919e39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp" Oct 30 18:18:57.352104 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:57.352064 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd45faeb-139e-4756-9b91-4ed08d919e39-metrics-certs\") pod \"maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp\" (UID: \"dd45faeb-139e-4756-9b91-4ed08d919e39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp" Oct 30 18:18:57.352300 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:57.352111 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd45faeb-139e-4756-9b91-4ed08d919e39-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp\" (UID: \"dd45faeb-139e-4756-9b91-4ed08d919e39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp" Oct 30 18:18:57.352300 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:57.352152 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd45faeb-139e-4756-9b91-4ed08d919e39-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp\" (UID: \"dd45faeb-139e-4756-9b91-4ed08d919e39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp" Oct 30 18:18:57.352300 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:57.352291 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js8zc\" (UniqueName: \"kubernetes.io/projected/dd45faeb-139e-4756-9b91-4ed08d919e39-kube-api-access-js8zc\") pod \"maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp\" (UID: \"dd45faeb-139e-4756-9b91-4ed08d919e39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp" Oct 30 18:18:57.352959 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:57.352932 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd45faeb-139e-4756-9b91-4ed08d919e39-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp\" (UID: \"dd45faeb-139e-4756-9b91-4ed08d919e39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp" Oct 30 18:18:57.354745 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:57.354717 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd45faeb-139e-4756-9b91-4ed08d919e39-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp\" (UID: \"dd45faeb-139e-4756-9b91-4ed08d919e39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp" Oct 30 18:18:57.354863 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:57.354783 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd45faeb-139e-4756-9b91-4ed08d919e39-metrics-certs\") pod \"maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp\" (UID: \"dd45faeb-139e-4756-9b91-4ed08d919e39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp" Oct 30 18:18:57.377444 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:57.377415 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js8zc\" (UniqueName: \"kubernetes.io/projected/dd45faeb-139e-4756-9b91-4ed08d919e39-kube-api-access-js8zc\") pod \"maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp\" (UID: \"dd45faeb-139e-4756-9b91-4ed08d919e39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp" Oct 30 18:18:57.539291 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:57.539258 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp" Oct 30 18:18:57.896456 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:57.896422 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp"] Oct 30 18:18:58.489708 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:58.489671 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp" event={"ID":"dd45faeb-139e-4756-9b91-4ed08d919e39","Type":"ContainerStarted","Data":"d16af3f8d3d132f6148ee5f98a780ef9d37055dba47040a6a030eacfb1338b2c"} Oct 30 18:18:58.490172 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:58.489732 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp" event={"ID":"dd45faeb-139e-4756-9b91-4ed08d919e39","Type":"ContainerStarted","Data":"4052feb8dbe0fcb71f831ab2c3a284e04b08a1c4b0a6f58c329275568bb1593e"} Oct 30 18:18:58.490172 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:58.489872 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp" Oct 30 18:18:58.546670 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:18:58.546564 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp" podStartSLOduration=1.162566778 podStartE2EDuration="1.546544547s" totalImagesPullingTime="383.977769ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:58.541732159 +0000 UTC m=+1591.029369193" watchObservedRunningTime="2026-10-30 18:18:58.546544547 +0000 UTC m=+1591.034181584" Oct 30 18:19:00.386473 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:19:00.386438 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:19:04.505802 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:04.505774 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp" Oct 30 18:19:12.425774 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:12.425740 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-86dcff4487-4j59n"] Oct 30 18:19:12.430525 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:12.430484 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-86dcff4487-4j59n" Oct 30 18:19:12.448413 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:12.448388 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a8dc70\"" Oct 30 18:19:12.499492 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:12.489034 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a8dc70-dockercfg-xkbth\"" Oct 30 18:19:12.554661 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:12.510411 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-86dcff4487-4j59n"] Oct 30 18:19:12.554661 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:12.533701 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb84843-d30a-48e9-83c6-82e5f4f6c33a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a8dc70-86dcff4487-4j59n\" (UID: \"2eb84843-d30a-48e9-83c6-82e5f4f6c33a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-86dcff4487-4j59n" Oct 30 18:19:12.554661 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:12.533891 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxqnp\" (UniqueName: \"kubernetes.io/projected/2eb84843-d30a-48e9-83c6-82e5f4f6c33a-kube-api-access-mxqnp\") pod \"payload-pre-processing-e2e-ipp-b-a8dc70-86dcff4487-4j59n\" (UID: \"2eb84843-d30a-48e9-83c6-82e5f4f6c33a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-86dcff4487-4j59n" Oct 30 18:19:12.647342 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:12.646521 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxqnp\" (UniqueName: \"kubernetes.io/projected/2eb84843-d30a-48e9-83c6-82e5f4f6c33a-kube-api-access-mxqnp\") pod \"payload-pre-processing-e2e-ipp-b-a8dc70-86dcff4487-4j59n\" (UID: \"2eb84843-d30a-48e9-83c6-82e5f4f6c33a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-86dcff4487-4j59n" Oct 30 18:19:12.647342 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:12.646864 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb84843-d30a-48e9-83c6-82e5f4f6c33a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a8dc70-86dcff4487-4j59n\" (UID: \"2eb84843-d30a-48e9-83c6-82e5f4f6c33a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-86dcff4487-4j59n" Oct 30 18:19:12.659268 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:12.647986 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb84843-d30a-48e9-83c6-82e5f4f6c33a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a8dc70-86dcff4487-4j59n\" (UID: \"2eb84843-d30a-48e9-83c6-82e5f4f6c33a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-86dcff4487-4j59n" Oct 30 18:19:12.722146 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:12.722094 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-d9fb4d884-4b2qs"] Oct 30 18:19:12.742374 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:12.740492 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-d9fb4d884-4b2qs" Oct 30 18:19:12.760343 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:12.745725 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-d9fb4d884-4b2qs"] Oct 30 18:19:12.760343 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:12.751252 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxqnp\" (UniqueName: \"kubernetes.io/projected/2eb84843-d30a-48e9-83c6-82e5f4f6c33a-kube-api-access-mxqnp\") pod \"payload-pre-processing-e2e-ipp-b-a8dc70-86dcff4487-4j59n\" (UID: \"2eb84843-d30a-48e9-83c6-82e5f4f6c33a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-86dcff4487-4j59n" Oct 30 18:19:12.853615 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:12.853573 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9v26\" (UniqueName: \"kubernetes.io/projected/6cf9453e-28f8-404b-a329-64d3b3398fb6-kube-api-access-c9v26\") pod \"payload-processing-e2e-ipp-b-a8dc70-d9fb4d884-4b2qs\" (UID: \"6cf9453e-28f8-404b-a329-64d3b3398fb6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-d9fb4d884-4b2qs" Oct 30 18:19:12.853943 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:12.853916 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cf9453e-28f8-404b-a329-64d3b3398fb6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a8dc70-d9fb4d884-4b2qs\" (UID: \"6cf9453e-28f8-404b-a329-64d3b3398fb6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-d9fb4d884-4b2qs" Oct 30 18:19:12.954802 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:12.954761 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cf9453e-28f8-404b-a329-64d3b3398fb6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a8dc70-d9fb4d884-4b2qs\" (UID: \"6cf9453e-28f8-404b-a329-64d3b3398fb6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-d9fb4d884-4b2qs" Oct 30 18:19:12.954990 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:12.954873 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9v26\" (UniqueName: \"kubernetes.io/projected/6cf9453e-28f8-404b-a329-64d3b3398fb6-kube-api-access-c9v26\") pod \"payload-processing-e2e-ipp-b-a8dc70-d9fb4d884-4b2qs\" (UID: \"6cf9453e-28f8-404b-a329-64d3b3398fb6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-d9fb4d884-4b2qs" Oct 30 18:19:12.955590 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:12.955571 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cf9453e-28f8-404b-a329-64d3b3398fb6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a8dc70-d9fb4d884-4b2qs\" (UID: \"6cf9453e-28f8-404b-a329-64d3b3398fb6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-d9fb4d884-4b2qs" Oct 30 18:19:12.982385 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:12.982283 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9v26\" (UniqueName: \"kubernetes.io/projected/6cf9453e-28f8-404b-a329-64d3b3398fb6-kube-api-access-c9v26\") pod \"payload-processing-e2e-ipp-b-a8dc70-d9fb4d884-4b2qs\" (UID: \"6cf9453e-28f8-404b-a329-64d3b3398fb6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-d9fb4d884-4b2qs" Oct 30 18:19:13.046101 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:13.046064 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-86dcff4487-4j59n" Oct 30 18:19:13.106027 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:13.105987 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-d9fb4d884-4b2qs" Oct 30 18:19:13.376206 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:19:13.376177 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb84843_d30a_48e9_83c6_82e5f4f6c33a.slice/crio-5d2d4d955d915eaa14dee7b4c329f4d3e2024e438bfe988ec3883d7e62b5000f WatchSource:0}: Error finding container 5d2d4d955d915eaa14dee7b4c329f4d3e2024e438bfe988ec3883d7e62b5000f: Status 404 returned error can't find the container with id 5d2d4d955d915eaa14dee7b4c329f4d3e2024e438bfe988ec3883d7e62b5000f Oct 30 18:19:13.376380 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:13.376287 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-86dcff4487-4j59n"] Oct 30 18:19:13.429692 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:19:13.429642 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cf9453e_28f8_404b_a329_64d3b3398fb6.slice/crio-f6777707ca37aa6e84f51bdeaa32fe6daa25fb861be2f8bdf5a3d93d1c5573ba WatchSource:0}: Error finding container f6777707ca37aa6e84f51bdeaa32fe6daa25fb861be2f8bdf5a3d93d1c5573ba: Status 404 returned error can't find the container with id f6777707ca37aa6e84f51bdeaa32fe6daa25fb861be2f8bdf5a3d93d1c5573ba Oct 30 18:19:13.431469 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:13.431442 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-d9fb4d884-4b2qs"] Oct 30 18:19:13.803953 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:13.803912 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-d9fb4d884-4b2qs" event={"ID":"6cf9453e-28f8-404b-a329-64d3b3398fb6","Type":"ContainerStarted","Data":"a6c037ded1088fb87560dc295febb851d5ebffa9fcbaf63af4d75cf42780618e"} Oct 30 18:19:13.804237 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:13.804218 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-d9fb4d884-4b2qs" event={"ID":"6cf9453e-28f8-404b-a329-64d3b3398fb6","Type":"ContainerStarted","Data":"f6777707ca37aa6e84f51bdeaa32fe6daa25fb861be2f8bdf5a3d93d1c5573ba"} Oct 30 18:19:13.805525 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:13.805477 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-d9fb4d884-4b2qs" Oct 30 18:19:13.810256 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:13.809345 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-86dcff4487-4j59n" event={"ID":"2eb84843-d30a-48e9-83c6-82e5f4f6c33a","Type":"ContainerStarted","Data":"832514224e69eef5c3e4da5415e229621eae4f2f133cc2655919d103bc4ac375"} Oct 30 18:19:13.810256 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:13.809394 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-86dcff4487-4j59n" event={"ID":"2eb84843-d30a-48e9-83c6-82e5f4f6c33a","Type":"ContainerStarted","Data":"5d2d4d955d915eaa14dee7b4c329f4d3e2024e438bfe988ec3883d7e62b5000f"} Oct 30 18:19:13.810256 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:13.810207 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-86dcff4487-4j59n" Oct 30 18:19:13.861696 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:13.861645 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-d9fb4d884-4b2qs" podStartSLOduration=1.861630741 podStartE2EDuration="1.861630741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:13.859522756 +0000 UTC m=+1606.347159791" watchObservedRunningTime="2026-10-30 18:19:13.861630741 +0000 UTC m=+1606.349267774" Oct 30 18:19:13.917521 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:13.917456 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-86dcff4487-4j59n" podStartSLOduration=1.917433369 podStartE2EDuration="1.917433369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:13.912651211 +0000 UTC m=+1606.400288247" watchObservedRunningTime="2026-10-30 18:19:13.917433369 +0000 UTC m=+1606.405070404" Oct 30 18:19:15.270146 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:15.270097 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-189cda-ffc578cd-gfkwj"] Oct 30 18:19:15.276117 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:15.276092 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-ffc578cd-gfkwj" Oct 30 18:19:15.296943 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:15.296912 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-189cda-ffc578cd-gfkwj"] Oct 30 18:19:15.303863 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:15.303833 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-189cda\"" Oct 30 18:19:15.345130 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:15.345094 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-fc765d467-mz9jx"] Oct 30 18:19:15.350560 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:15.350540 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-fc765d467-mz9jx" Oct 30 18:19:15.363069 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:15.363044 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-fc765d467-mz9jx"] Oct 30 18:19:15.367105 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:15.367062 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-189cda-dockercfg-bgd9p\"" Oct 30 18:19:15.507346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:15.445198 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b83753-e4ee-4515-af11-8c8774fee80e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-189cda-fc765d467-mz9jx\" (UID: \"b6b83753-e4ee-4515-af11-8c8774fee80e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-fc765d467-mz9jx" Oct 30 18:19:15.507346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:15.445313 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvvhv\" (UniqueName: \"kubernetes.io/projected/af2d0e78-3caa-455d-8f94-701e396a9324-kube-api-access-zvvhv\") pod \"payload-processing-e2e-ipp-b-189cda-ffc578cd-gfkwj\" (UID: \"af2d0e78-3caa-455d-8f94-701e396a9324\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-ffc578cd-gfkwj" Oct 30 18:19:15.507346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:15.445451 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af2d0e78-3caa-455d-8f94-701e396a9324-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-189cda-ffc578cd-gfkwj\" (UID: \"af2d0e78-3caa-455d-8f94-701e396a9324\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-ffc578cd-gfkwj" Oct 30 18:19:15.507346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:15.445607 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qqjn\" (UniqueName: \"kubernetes.io/projected/b6b83753-e4ee-4515-af11-8c8774fee80e-kube-api-access-5qqjn\") pod \"payload-pre-processing-e2e-ipp-b-189cda-fc765d467-mz9jx\" (UID: \"b6b83753-e4ee-4515-af11-8c8774fee80e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-fc765d467-mz9jx" Oct 30 18:19:15.603405 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:15.560640 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qqjn\" (UniqueName: \"kubernetes.io/projected/b6b83753-e4ee-4515-af11-8c8774fee80e-kube-api-access-5qqjn\") pod \"payload-pre-processing-e2e-ipp-b-189cda-fc765d467-mz9jx\" (UID: \"b6b83753-e4ee-4515-af11-8c8774fee80e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-fc765d467-mz9jx" Oct 30 18:19:15.603405 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:15.560758 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b83753-e4ee-4515-af11-8c8774fee80e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-189cda-fc765d467-mz9jx\" (UID: \"b6b83753-e4ee-4515-af11-8c8774fee80e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-fc765d467-mz9jx" Oct 30 18:19:15.603405 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:15.560863 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvvhv\" (UniqueName: \"kubernetes.io/projected/af2d0e78-3caa-455d-8f94-701e396a9324-kube-api-access-zvvhv\") pod \"payload-processing-e2e-ipp-b-189cda-ffc578cd-gfkwj\" (UID: \"af2d0e78-3caa-455d-8f94-701e396a9324\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-ffc578cd-gfkwj" Oct 30 18:19:15.603405 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:15.560985 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af2d0e78-3caa-455d-8f94-701e396a9324-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-189cda-ffc578cd-gfkwj\" (UID: \"af2d0e78-3caa-455d-8f94-701e396a9324\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-ffc578cd-gfkwj" Oct 30 18:19:15.603405 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:15.562015 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b83753-e4ee-4515-af11-8c8774fee80e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-189cda-fc765d467-mz9jx\" (UID: \"b6b83753-e4ee-4515-af11-8c8774fee80e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-fc765d467-mz9jx" Oct 30 18:19:15.603405 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:15.587385 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af2d0e78-3caa-455d-8f94-701e396a9324-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-189cda-ffc578cd-gfkwj\" (UID: \"af2d0e78-3caa-455d-8f94-701e396a9324\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-ffc578cd-gfkwj" Oct 30 18:19:15.709917 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:15.709883 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvvhv\" (UniqueName: \"kubernetes.io/projected/af2d0e78-3caa-455d-8f94-701e396a9324-kube-api-access-zvvhv\") pod \"payload-processing-e2e-ipp-b-189cda-ffc578cd-gfkwj\" (UID: \"af2d0e78-3caa-455d-8f94-701e396a9324\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-ffc578cd-gfkwj" Oct 30 18:19:15.720157 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:15.714718 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qqjn\" (UniqueName: \"kubernetes.io/projected/b6b83753-e4ee-4515-af11-8c8774fee80e-kube-api-access-5qqjn\") pod \"payload-pre-processing-e2e-ipp-b-189cda-fc765d467-mz9jx\" (UID: \"b6b83753-e4ee-4515-af11-8c8774fee80e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-fc765d467-mz9jx" Oct 30 18:19:15.894865 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:15.894756 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-ffc578cd-gfkwj" Oct 30 18:19:15.969382 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:15.969142 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-fc765d467-mz9jx" Oct 30 18:19:16.228645 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:19:16.228614 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf2d0e78_3caa_455d_8f94_701e396a9324.slice/crio-c3d6cef86bf96a8ebfb7720a2a9aee5228009d29d8dd96f196ddc91013addd76 WatchSource:0}: Error finding container c3d6cef86bf96a8ebfb7720a2a9aee5228009d29d8dd96f196ddc91013addd76: Status 404 returned error can't find the container with id c3d6cef86bf96a8ebfb7720a2a9aee5228009d29d8dd96f196ddc91013addd76 Oct 30 18:19:16.232661 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:16.232636 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-189cda-ffc578cd-gfkwj"] Oct 30 18:19:16.420631 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:19:16.420597 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b83753_e4ee_4515_af11_8c8774fee80e.slice/crio-1ca418a34f86cbf7e28bd3fd0482fccf1c40f40b60fe87c8eb7390229c0054c6 WatchSource:0}: Error finding container 1ca418a34f86cbf7e28bd3fd0482fccf1c40f40b60fe87c8eb7390229c0054c6: Status 404 returned error can't find the container with id 1ca418a34f86cbf7e28bd3fd0482fccf1c40f40b60fe87c8eb7390229c0054c6 Oct 30 18:19:16.422639 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:16.422610 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-fc765d467-mz9jx"] Oct 30 18:19:16.905176 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:16.905134 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-fc765d467-mz9jx" event={"ID":"b6b83753-e4ee-4515-af11-8c8774fee80e","Type":"ContainerStarted","Data":"09eb4f86b0c59f81da73fbd2cd8e010f90cbf53a8f4d8f859849ecfa2fea9c2e"} Oct 30 18:19:16.905506 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:16.905476 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-fc765d467-mz9jx" event={"ID":"b6b83753-e4ee-4515-af11-8c8774fee80e","Type":"ContainerStarted","Data":"1ca418a34f86cbf7e28bd3fd0482fccf1c40f40b60fe87c8eb7390229c0054c6"} Oct 30 18:19:16.908690 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:16.908653 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-ffc578cd-gfkwj" event={"ID":"af2d0e78-3caa-455d-8f94-701e396a9324","Type":"ContainerStarted","Data":"9793883a40a127118495dfb68ded0810b7c37fa009dbffd24893a43458692d3f"} Oct 30 18:19:16.908933 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:16.908911 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-ffc578cd-gfkwj" event={"ID":"af2d0e78-3caa-455d-8f94-701e396a9324","Type":"ContainerStarted","Data":"c3d6cef86bf96a8ebfb7720a2a9aee5228009d29d8dd96f196ddc91013addd76"} Oct 30 18:19:16.910422 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:16.910403 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-ffc578cd-gfkwj" Oct 30 18:19:16.911359 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:16.911343 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-fc765d467-mz9jx" Oct 30 18:19:16.958856 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:16.958801 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-ffc578cd-gfkwj" podStartSLOduration=1.9587859619999999 podStartE2EDuration="1.958785962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:16.953673959 +0000 UTC m=+1609.441310994" watchObservedRunningTime="2026-10-30 18:19:16.958785962 +0000 UTC m=+1609.446422996" Oct 30 18:19:17.006393 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:17.006308 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-fc765d467-mz9jx" podStartSLOduration=2.006289149 podStartE2EDuration="2.006289149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:17.003892617 +0000 UTC m=+1609.491529653" watchObservedRunningTime="2026-10-30 18:19:17.006289149 +0000 UTC m=+1609.493926183" Oct 30 18:19:24.219699 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:24.219660 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf"] Oct 30 18:19:24.220267 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:24.219941 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" podUID="87894ec9-1772-4a1d-9b7d-e6489df743a4" containerName="istio-proxy" containerID="cri-o://c8384c9571fc8961ee5feb5c0f55c39442abf65ca37b0b306e5a5e3e33490a5f" gracePeriod=30 Oct 30 18:19:25.879347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:25.869625 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-86dcff4487-4j59n" Oct 30 18:19:25.880372 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:25.880341 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-d9fb4d884-4b2qs" Oct 30 18:19:29.025383 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:29.025356 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-fc765d467-mz9jx" Oct 30 18:19:29.025868 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:29.025546 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-ffc578cd-gfkwj" Oct 30 18:19:29.626499 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:29.626472 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" Oct 30 18:19:29.695455 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:29.695351 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87894ec9-1772-4a1d-9b7d-e6489df743a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87894ec9-1772-4a1d-9b7d-e6489df743a4-istio-envoy\") pod \"87894ec9-1772-4a1d-9b7d-e6489df743a4\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " Oct 30 18:19:29.695455 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:29.695436 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87894ec9-1772-4a1d-9b7d-e6489df743a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87894ec9-1772-4a1d-9b7d-e6489df743a4-istiod-ca-cert\") pod \"87894ec9-1772-4a1d-9b7d-e6489df743a4\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " Oct 30 18:19:29.695713 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:29.695575 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87894ec9-1772-4a1d-9b7d-e6489df743a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87894ec9-1772-4a1d-9b7d-e6489df743a4-workload-socket\") pod \"87894ec9-1772-4a1d-9b7d-e6489df743a4\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " Oct 30 18:19:29.695713 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:29.695626 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87894ec9-1772-4a1d-9b7d-e6489df743a4-kube-api-access-6pg5d\" (UniqueName: \"kubernetes.io/projected/87894ec9-1772-4a1d-9b7d-e6489df743a4-kube-api-access-6pg5d\") pod \"87894ec9-1772-4a1d-9b7d-e6489df743a4\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " Oct 30 18:19:29.695831 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:29.695761 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87894ec9-1772-4a1d-9b7d-e6489df743a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87894ec9-1772-4a1d-9b7d-e6489df743a4-credential-socket\") pod \"87894ec9-1772-4a1d-9b7d-e6489df743a4\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " Oct 30 18:19:29.695944 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:29.695922 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87894ec9-1772-4a1d-9b7d-e6489df743a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87894ec9-1772-4a1d-9b7d-e6489df743a4-workload-certs\") pod \"87894ec9-1772-4a1d-9b7d-e6489df743a4\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " Oct 30 18:19:29.696015 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:29.695970 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87894ec9-1772-4a1d-9b7d-e6489df743a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87894ec9-1772-4a1d-9b7d-e6489df743a4-istio-podinfo\") pod \"87894ec9-1772-4a1d-9b7d-e6489df743a4\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " Oct 30 18:19:29.696183 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:29.696127 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87894ec9-1772-4a1d-9b7d-e6489df743a4-istio-token\" (UniqueName: \"kubernetes.io/projected/87894ec9-1772-4a1d-9b7d-e6489df743a4-istio-token\") pod \"87894ec9-1772-4a1d-9b7d-e6489df743a4\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " Oct 30 18:19:29.696267 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:29.696210 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87894ec9-1772-4a1d-9b7d-e6489df743a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87894ec9-1772-4a1d-9b7d-e6489df743a4-istio-data\") pod \"87894ec9-1772-4a1d-9b7d-e6489df743a4\" (UID: \"87894ec9-1772-4a1d-9b7d-e6489df743a4\") " Oct 30 18:19:29.696573 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:29.696545 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87894ec9-1772-4a1d-9b7d-e6489df743a4-workload-certs" pod "87894ec9-1772-4a1d-9b7d-e6489df743a4" (UID: "87894ec9-1772-4a1d-9b7d-e6489df743a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:29.696792 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:29.696770 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87894ec9-1772-4a1d-9b7d-e6489df743a4-credential-socket" pod "87894ec9-1772-4a1d-9b7d-e6489df743a4" (UID: "87894ec9-1772-4a1d-9b7d-e6489df743a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:29.696955 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:29.696932 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87894ec9-1772-4a1d-9b7d-e6489df743a4-istiod-ca-cert" pod "87894ec9-1772-4a1d-9b7d-e6489df743a4" (UID: "87894ec9-1772-4a1d-9b7d-e6489df743a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:29.697209 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:29.697184 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87894ec9-1772-4a1d-9b7d-e6489df743a4-workload-socket" pod "87894ec9-1772-4a1d-9b7d-e6489df743a4" (UID: "87894ec9-1772-4a1d-9b7d-e6489df743a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:29.698068 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:29.698040 4455 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87894ec9-1772-4a1d-9b7d-e6489df743a4-istiod-ca-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.698151 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:29.698074 4455 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87894ec9-1772-4a1d-9b7d-e6489df743a4-workload-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.698151 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:29.698088 4455 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87894ec9-1772-4a1d-9b7d-e6489df743a4-credential-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.698151 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:29.698101 4455 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87894ec9-1772-4a1d-9b7d-e6489df743a4-workload-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.700149 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:29.700043 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87894ec9-1772-4a1d-9b7d-e6489df743a4-istio-envoy" pod "87894ec9-1772-4a1d-9b7d-e6489df743a4" (UID: "87894ec9-1772-4a1d-9b7d-e6489df743a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:29.701196 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:29.700915 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87894ec9-1772-4a1d-9b7d-e6489df743a4-istio-token" pod "87894ec9-1772-4a1d-9b7d-e6489df743a4" (UID: "87894ec9-1772-4a1d-9b7d-e6489df743a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:29.701603 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:29.701583 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87894ec9-1772-4a1d-9b7d-e6489df743a4-kube-api-access-6pg5d" pod "87894ec9-1772-4a1d-9b7d-e6489df743a4" (UID: "87894ec9-1772-4a1d-9b7d-e6489df743a4"). InnerVolumeSpecName "kube-api-access-6pg5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:29.701691 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:29.701645 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87894ec9-1772-4a1d-9b7d-e6489df743a4-istio-data" pod "87894ec9-1772-4a1d-9b7d-e6489df743a4" (UID: "87894ec9-1772-4a1d-9b7d-e6489df743a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:29.703160 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:29.703139 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87894ec9-1772-4a1d-9b7d-e6489df743a4-istio-podinfo" pod "87894ec9-1772-4a1d-9b7d-e6489df743a4" (UID: "87894ec9-1772-4a1d-9b7d-e6489df743a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:19:29.799899 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:29.799858 4455 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87894ec9-1772-4a1d-9b7d-e6489df743a4-istio-podinfo\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.799899 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:29.799891 4455 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87894ec9-1772-4a1d-9b7d-e6489df743a4-istio-token\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.799899 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:29.799902 4455 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87894ec9-1772-4a1d-9b7d-e6489df743a4-istio-data\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.800143 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:29.799912 4455 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87894ec9-1772-4a1d-9b7d-e6489df743a4-istio-envoy\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.800143 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:29.799927 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pg5d\" (UniqueName: \"kubernetes.io/projected/87894ec9-1772-4a1d-9b7d-e6489df743a4-kube-api-access-6pg5d\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:19:30.174629 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:30.174595 4455 generic.go:356] "Generic (PLEG): container finished" podID="87894ec9-1772-4a1d-9b7d-e6489df743a4" containerID="c8384c9571fc8961ee5feb5c0f55c39442abf65ca37b0b306e5a5e3e33490a5f" exitCode=0 Oct 30 18:19:30.175146 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:30.174640 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" Oct 30 18:19:30.175146 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:30.174682 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" event={"ID":"87894ec9-1772-4a1d-9b7d-e6489df743a4","Type":"ContainerDied","Data":"c8384c9571fc8961ee5feb5c0f55c39442abf65ca37b0b306e5a5e3e33490a5f"} Oct 30 18:19:30.175146 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:30.174740 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf" event={"ID":"87894ec9-1772-4a1d-9b7d-e6489df743a4","Type":"ContainerDied","Data":"0387748736b163530244a52dc4abfcdeb98a5a0676dd29918fe2ae1ba0d8067e"} Oct 30 18:19:30.175146 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:30.174764 4455 scope.go:122] "RemoveContainer" containerID="c8384c9571fc8961ee5feb5c0f55c39442abf65ca37b0b306e5a5e3e33490a5f" Oct 30 18:19:30.187708 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:30.187688 4455 scope.go:122] "RemoveContainer" containerID="c8384c9571fc8961ee5feb5c0f55c39442abf65ca37b0b306e5a5e3e33490a5f" Oct 30 18:19:30.188095 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:19:30.188070 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8384c9571fc8961ee5feb5c0f55c39442abf65ca37b0b306e5a5e3e33490a5f\": container with ID starting with c8384c9571fc8961ee5feb5c0f55c39442abf65ca37b0b306e5a5e3e33490a5f not found: ID does not exist" containerID="c8384c9571fc8961ee5feb5c0f55c39442abf65ca37b0b306e5a5e3e33490a5f" Oct 30 18:19:30.188200 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:30.188102 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8384c9571fc8961ee5feb5c0f55c39442abf65ca37b0b306e5a5e3e33490a5f"} err="failed to get container status \"c8384c9571fc8961ee5feb5c0f55c39442abf65ca37b0b306e5a5e3e33490a5f\": rpc error: code = NotFound desc = could not find container \"c8384c9571fc8961ee5feb5c0f55c39442abf65ca37b0b306e5a5e3e33490a5f\": container with ID starting with c8384c9571fc8961ee5feb5c0f55c39442abf65ca37b0b306e5a5e3e33490a5f not found: ID does not exist" Oct 30 18:19:30.233895 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:30.233822 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf"] Oct 30 18:19:30.247299 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:30.247266 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-wk6xf"] Oct 30 18:19:30.387416 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:19:30.387380 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:19:31.243732 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:31.243662 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a3be60-cd57b57d7-fnl2f"] Oct 30 18:19:31.244639 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:31.244614 4455 cpu_manager.go:414] "containerMap: removing container" podUID="87894ec9-1772-4a1d-9b7d-e6489df743a4" containerName="istio-proxy" Oct 30 18:19:31.244639 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:31.244638 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87894ec9-1772-4a1d-9b7d-e6489df743a4" containerName="istio-proxy" Oct 30 18:19:31.244836 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:31.244798 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="87894ec9-1772-4a1d-9b7d-e6489df743a4" containerName="istio-proxy" Oct 30 18:19:31.250519 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:31.250489 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc"] Oct 30 18:19:31.250640 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:31.250615 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3be60-cd57b57d7-fnl2f" Oct 30 18:19:31.255182 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:31.255158 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc" Oct 30 18:19:31.267752 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:31.267714 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a3be60\"" Oct 30 18:19:31.276523 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:31.276484 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a3be60-dockercfg-9658c\"" Oct 30 18:19:31.280421 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:31.278452 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a8dc70-serving-cert\"" Oct 30 18:19:31.306198 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:31.306153 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a3be60-cd57b57d7-fnl2f"] Oct 30 18:19:31.309627 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:31.309594 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc"] Oct 30 18:19:31.323461 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:31.323427 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e084cf95-22eb-4209-84a6-9c942327cbd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a3be60-cd57b57d7-fnl2f\" (UID: \"e084cf95-22eb-4209-84a6-9c942327cbd7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3be60-cd57b57d7-fnl2f" Oct 30 18:19:31.323641 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:31.323500 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55217532-8833-4f27-92cf-509c8059409b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc\" (UID: \"55217532-8833-4f27-92cf-509c8059409b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc" Oct 30 18:19:31.323641 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:31.323592 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sr2c\" (UniqueName: \"kubernetes.io/projected/55217532-8833-4f27-92cf-509c8059409b-kube-api-access-5sr2c\") pod \"maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc\" (UID: \"55217532-8833-4f27-92cf-509c8059409b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc" Oct 30 18:19:31.323641 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:31.323634 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgp9p\" (UniqueName: \"kubernetes.io/projected/e084cf95-22eb-4209-84a6-9c942327cbd7-kube-api-access-vgp9p\") pod \"payload-pre-processing-e2e-api-a-a3be60-cd57b57d7-fnl2f\" (UID: \"e084cf95-22eb-4209-84a6-9c942327cbd7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3be60-cd57b57d7-fnl2f" Oct 30 18:19:31.323828 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:31.323667 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55217532-8833-4f27-92cf-509c8059409b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc\" (UID: \"55217532-8833-4f27-92cf-509c8059409b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc" Oct 30 18:19:31.323828 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:31.323715 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55217532-8833-4f27-92cf-509c8059409b-metrics-certs\") pod \"maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc\" (UID: \"55217532-8833-4f27-92cf-509c8059409b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc" Oct 30 18:19:31.424771 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:31.424738 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e084cf95-22eb-4209-84a6-9c942327cbd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a3be60-cd57b57d7-fnl2f\" (UID: \"e084cf95-22eb-4209-84a6-9c942327cbd7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3be60-cd57b57d7-fnl2f" Oct 30 18:19:31.424975 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:31.424815 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55217532-8833-4f27-92cf-509c8059409b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc\" (UID: \"55217532-8833-4f27-92cf-509c8059409b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc" Oct 30 18:19:31.424975 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:31.424907 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sr2c\" (UniqueName: \"kubernetes.io/projected/55217532-8833-4f27-92cf-509c8059409b-kube-api-access-5sr2c\") pod \"maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc\" (UID: \"55217532-8833-4f27-92cf-509c8059409b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc" Oct 30 18:19:31.424975 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:31.424949 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgp9p\" (UniqueName: \"kubernetes.io/projected/e084cf95-22eb-4209-84a6-9c942327cbd7-kube-api-access-vgp9p\") pod \"payload-pre-processing-e2e-api-a-a3be60-cd57b57d7-fnl2f\" (UID: \"e084cf95-22eb-4209-84a6-9c942327cbd7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3be60-cd57b57d7-fnl2f" Oct 30 18:19:31.425149 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:19:31.424978 4455 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8dc70-serving-cert: secret "maas-api-e2e-ipp-b-a8dc70-serving-cert" not found Oct 30 18:19:31.425149 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:19:31.425053 4455 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55217532-8833-4f27-92cf-509c8059409b-maas-api-tls podName:55217532-8833-4f27-92cf-509c8059409b nodeName:}" failed. No retries permitted until 2026-10-30 18:19:31.925031512 +0000 UTC m=+1624.412668538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/55217532-8833-4f27-92cf-509c8059409b-maas-api-tls") pod "maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc" (UID: "55217532-8833-4f27-92cf-509c8059409b") : secret "maas-api-e2e-ipp-b-a8dc70-serving-cert" not found Oct 30 18:19:31.425303 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:31.424989 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55217532-8833-4f27-92cf-509c8059409b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc\" (UID: \"55217532-8833-4f27-92cf-509c8059409b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc" Oct 30 18:19:31.425410 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:31.425383 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55217532-8833-4f27-92cf-509c8059409b-metrics-certs\") pod \"maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc\" (UID: \"55217532-8833-4f27-92cf-509c8059409b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc" Oct 30 18:19:31.425684 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:31.425658 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55217532-8833-4f27-92cf-509c8059409b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc\" (UID: \"55217532-8833-4f27-92cf-509c8059409b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc" Oct 30 18:19:31.425772 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:31.425698 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e084cf95-22eb-4209-84a6-9c942327cbd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a3be60-cd57b57d7-fnl2f\" (UID: \"e084cf95-22eb-4209-84a6-9c942327cbd7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3be60-cd57b57d7-fnl2f" Oct 30 18:19:31.428371 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:31.428304 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55217532-8833-4f27-92cf-509c8059409b-metrics-certs\") pod \"maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc\" (UID: \"55217532-8833-4f27-92cf-509c8059409b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc" Oct 30 18:19:31.614390 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:31.608378 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sr2c\" (UniqueName: \"kubernetes.io/projected/55217532-8833-4f27-92cf-509c8059409b-kube-api-access-5sr2c\") pod \"maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc\" (UID: \"55217532-8833-4f27-92cf-509c8059409b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc" Oct 30 18:19:31.654352 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:31.653751 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgp9p\" (UniqueName: \"kubernetes.io/projected/e084cf95-22eb-4209-84a6-9c942327cbd7-kube-api-access-vgp9p\") pod \"payload-pre-processing-e2e-api-a-a3be60-cd57b57d7-fnl2f\" (UID: \"e084cf95-22eb-4209-84a6-9c942327cbd7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3be60-cd57b57d7-fnl2f" Oct 30 18:19:31.678350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:31.667021 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87894ec9-1772-4a1d-9b7d-e6489df743a4" path="/var/lib/kubelet/pods/87894ec9-1772-4a1d-9b7d-e6489df743a4/volumes" Oct 30 18:19:31.921126 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:31.921032 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3be60-cd57b57d7-fnl2f" Oct 30 18:19:31.968343 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:31.968284 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55217532-8833-4f27-92cf-509c8059409b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc\" (UID: \"55217532-8833-4f27-92cf-509c8059409b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc" Oct 30 18:19:31.972165 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:31.972132 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55217532-8833-4f27-92cf-509c8059409b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc\" (UID: \"55217532-8833-4f27-92cf-509c8059409b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc" Oct 30 18:19:32.232395 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:32.232310 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc" Oct 30 18:19:32.262169 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:19:32.262133 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode084cf95_22eb_4209_84a6_9c942327cbd7.slice/crio-0d38cfcfdf806ee0574e648ceb559b4fd290ddb242499d282457ce7978558521 WatchSource:0}: Error finding container 0d38cfcfdf806ee0574e648ceb559b4fd290ddb242499d282457ce7978558521: Status 404 returned error can't find the container with id 0d38cfcfdf806ee0574e648ceb559b4fd290ddb242499d282457ce7978558521 Oct 30 18:19:32.266003 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:32.265959 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a3be60-cd57b57d7-fnl2f"] Oct 30 18:19:32.616544 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:32.616446 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc"] Oct 30 18:19:33.241197 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:33.241160 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3be60-cd57b57d7-fnl2f" event={"ID":"e084cf95-22eb-4209-84a6-9c942327cbd7","Type":"ContainerStarted","Data":"93fad3ac3046816909627da35cd9ec88ae9a573d412d400de3539e4fae833d37"} Oct 30 18:19:33.241448 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:33.241223 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3be60-cd57b57d7-fnl2f" event={"ID":"e084cf95-22eb-4209-84a6-9c942327cbd7","Type":"ContainerStarted","Data":"0d38cfcfdf806ee0574e648ceb559b4fd290ddb242499d282457ce7978558521"} Oct 30 18:19:33.241448 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:33.241257 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3be60-cd57b57d7-fnl2f" Oct 30 18:19:33.243078 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:33.243051 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc" event={"ID":"55217532-8833-4f27-92cf-509c8059409b","Type":"ContainerStarted","Data":"6c1a97f14e7ca3198a55c13f916db9c2c8d3154bb32c021bd1f1523bab26bde9"} Oct 30 18:19:33.243212 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:33.243107 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc" event={"ID":"55217532-8833-4f27-92cf-509c8059409b","Type":"ContainerStarted","Data":"fa980990e4e6162d18ef6231e9235f3803ee44b2bfa42dc0f228e5d99974d0e1"} Oct 30 18:19:33.243212 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:33.243134 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc" Oct 30 18:19:33.346042 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:33.345913 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3be60-cd57b57d7-fnl2f" podStartSLOduration=2.345888549 podStartE2EDuration="2.345888549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:33.297729434 +0000 UTC m=+1625.785366469" watchObservedRunningTime="2026-10-30 18:19:33.345888549 +0000 UTC m=+1625.833525585" Oct 30 18:19:33.373351 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:33.371727 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc" podStartSLOduration=1.947782828 podStartE2EDuration="2.371710803s" totalImagesPullingTime="423.927975ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:33.371623581 +0000 UTC m=+1625.859260617" watchObservedRunningTime="2026-10-30 18:19:33.371710803 +0000 UTC m=+1625.859347838" Oct 30 18:19:35.489910 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:35.489866 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q"] Oct 30 18:19:35.501426 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:35.501389 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q" Oct 30 18:19:35.522732 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:35.522698 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q"] Oct 30 18:19:35.586378 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:35.572396 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-189cda-serving-cert\"" Oct 30 18:19:35.592298 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:35.592256 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwmvz\" (UniqueName: \"kubernetes.io/projected/f4cf887a-fbe0-4738-b919-5110b2f77a7b-kube-api-access-fwmvz\") pod \"maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q\" (UID: \"f4cf887a-fbe0-4738-b919-5110b2f77a7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q" Oct 30 18:19:35.592454 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:35.592413 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4cf887a-fbe0-4738-b919-5110b2f77a7b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q\" (UID: \"f4cf887a-fbe0-4738-b919-5110b2f77a7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q" Oct 30 18:19:35.592454 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:35.592437 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4cf887a-fbe0-4738-b919-5110b2f77a7b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q\" (UID: \"f4cf887a-fbe0-4738-b919-5110b2f77a7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q" Oct 30 18:19:35.592588 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:35.592575 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4cf887a-fbe0-4738-b919-5110b2f77a7b-metrics-certs\") pod \"maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q\" (UID: \"f4cf887a-fbe0-4738-b919-5110b2f77a7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q" Oct 30 18:19:35.694619 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:35.694579 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4cf887a-fbe0-4738-b919-5110b2f77a7b-metrics-certs\") pod \"maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q\" (UID: \"f4cf887a-fbe0-4738-b919-5110b2f77a7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q" Oct 30 18:19:35.694843 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:35.694814 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwmvz\" (UniqueName: \"kubernetes.io/projected/f4cf887a-fbe0-4738-b919-5110b2f77a7b-kube-api-access-fwmvz\") pod \"maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q\" (UID: \"f4cf887a-fbe0-4738-b919-5110b2f77a7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q" Oct 30 18:19:35.694920 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:35.694903 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4cf887a-fbe0-4738-b919-5110b2f77a7b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q\" (UID: \"f4cf887a-fbe0-4738-b919-5110b2f77a7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q" Oct 30 18:19:35.694983 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:35.694935 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4cf887a-fbe0-4738-b919-5110b2f77a7b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q\" (UID: \"f4cf887a-fbe0-4738-b919-5110b2f77a7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q" Oct 30 18:19:35.696596 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:35.696565 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4cf887a-fbe0-4738-b919-5110b2f77a7b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q\" (UID: \"f4cf887a-fbe0-4738-b919-5110b2f77a7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q" Oct 30 18:19:35.697743 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:35.697723 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4cf887a-fbe0-4738-b919-5110b2f77a7b-metrics-certs\") pod \"maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q\" (UID: \"f4cf887a-fbe0-4738-b919-5110b2f77a7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q" Oct 30 18:19:35.697855 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:35.697840 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4cf887a-fbe0-4738-b919-5110b2f77a7b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q\" (UID: \"f4cf887a-fbe0-4738-b919-5110b2f77a7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q" Oct 30 18:19:35.720315 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:35.720288 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwmvz\" (UniqueName: \"kubernetes.io/projected/f4cf887a-fbe0-4738-b919-5110b2f77a7b-kube-api-access-fwmvz\") pod \"maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q\" (UID: \"f4cf887a-fbe0-4738-b919-5110b2f77a7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q" Oct 30 18:19:35.828975 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:35.828626 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q" Oct 30 18:19:36.205973 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:19:36.205938 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4cf887a_fbe0_4738_b919_5110b2f77a7b.slice/crio-1b2452f8d8a3a9e4f0c4f0ad4cef0cea7724f997272b1d4c6c168f5f91fb8da1 WatchSource:0}: Error finding container 1b2452f8d8a3a9e4f0c4f0ad4cef0cea7724f997272b1d4c6c168f5f91fb8da1: Status 404 returned error can't find the container with id 1b2452f8d8a3a9e4f0c4f0ad4cef0cea7724f997272b1d4c6c168f5f91fb8da1 Oct 30 18:19:36.206239 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:36.206215 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q"] Oct 30 18:19:36.266880 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:36.266843 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q" event={"ID":"f4cf887a-fbe0-4738-b919-5110b2f77a7b","Type":"ContainerStarted","Data":"1b2452f8d8a3a9e4f0c4f0ad4cef0cea7724f997272b1d4c6c168f5f91fb8da1"} Oct 30 18:19:37.314730 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:37.309612 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q" event={"ID":"f4cf887a-fbe0-4738-b919-5110b2f77a7b","Type":"ContainerStarted","Data":"e41ef75fae079c585b0ccffdf63aaf0ebb7517e390219fd5d5eb359124415c74"} Oct 30 18:19:37.314730 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:37.310026 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q" Oct 30 18:19:39.256127 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:39.256095 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc" Oct 30 18:19:39.302310 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:39.302243 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q" podStartSLOduration=3.903610905 podStartE2EDuration="4.302224358s" totalImagesPullingTime="398.613453ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:37.40518272 +0000 UTC m=+1629.892819754" watchObservedRunningTime="2026-10-30 18:19:39.302224358 +0000 UTC m=+1631.789861393" Oct 30 18:19:43.327828 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:43.327794 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q" Oct 30 18:19:44.254176 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:44.254146 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3be60-cd57b57d7-fnl2f" Oct 30 18:19:52.399219 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:52.399181 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t"] Oct 30 18:19:52.404063 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:52.404039 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t" Oct 30 18:19:52.427433 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:52.427403 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t"] Oct 30 18:19:52.444241 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:52.435016 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a3be60-serving-cert\"" Oct 30 18:19:52.453565 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:52.453529 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ef482c4-f1c6-4b06-8e14-220e622fa711-openshift-service-ca\") pod \"maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t\" (UID: \"7ef482c4-f1c6-4b06-8e14-220e622fa711\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t" Oct 30 18:19:52.453918 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:52.453894 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lpsm\" (UniqueName: \"kubernetes.io/projected/7ef482c4-f1c6-4b06-8e14-220e622fa711-kube-api-access-8lpsm\") pod \"maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t\" (UID: \"7ef482c4-f1c6-4b06-8e14-220e622fa711\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t" Oct 30 18:19:52.454052 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:52.453998 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ef482c4-f1c6-4b06-8e14-220e622fa711-maas-api-tls\") pod \"maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t\" (UID: \"7ef482c4-f1c6-4b06-8e14-220e622fa711\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t" Oct 30 18:19:52.454052 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:52.454047 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ef482c4-f1c6-4b06-8e14-220e622fa711-metrics-certs\") pod \"maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t\" (UID: \"7ef482c4-f1c6-4b06-8e14-220e622fa711\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t" Oct 30 18:19:52.555599 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:52.555556 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lpsm\" (UniqueName: \"kubernetes.io/projected/7ef482c4-f1c6-4b06-8e14-220e622fa711-kube-api-access-8lpsm\") pod \"maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t\" (UID: \"7ef482c4-f1c6-4b06-8e14-220e622fa711\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t" Oct 30 18:19:52.555793 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:52.555662 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ef482c4-f1c6-4b06-8e14-220e622fa711-maas-api-tls\") pod \"maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t\" (UID: \"7ef482c4-f1c6-4b06-8e14-220e622fa711\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t" Oct 30 18:19:52.555793 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:52.555711 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ef482c4-f1c6-4b06-8e14-220e622fa711-metrics-certs\") pod \"maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t\" (UID: \"7ef482c4-f1c6-4b06-8e14-220e622fa711\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t" Oct 30 18:19:52.555914 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:52.555885 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ef482c4-f1c6-4b06-8e14-220e622fa711-openshift-service-ca\") pod \"maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t\" (UID: \"7ef482c4-f1c6-4b06-8e14-220e622fa711\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t" Oct 30 18:19:52.556588 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:52.556561 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ef482c4-f1c6-4b06-8e14-220e622fa711-openshift-service-ca\") pod \"maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t\" (UID: \"7ef482c4-f1c6-4b06-8e14-220e622fa711\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t" Oct 30 18:19:52.558239 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:52.558213 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ef482c4-f1c6-4b06-8e14-220e622fa711-maas-api-tls\") pod \"maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t\" (UID: \"7ef482c4-f1c6-4b06-8e14-220e622fa711\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t" Oct 30 18:19:52.558450 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:52.558431 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ef482c4-f1c6-4b06-8e14-220e622fa711-metrics-certs\") pod \"maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t\" (UID: \"7ef482c4-f1c6-4b06-8e14-220e622fa711\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t" Oct 30 18:19:52.584876 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:52.584835 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lpsm\" (UniqueName: \"kubernetes.io/projected/7ef482c4-f1c6-4b06-8e14-220e622fa711-kube-api-access-8lpsm\") pod \"maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t\" (UID: \"7ef482c4-f1c6-4b06-8e14-220e622fa711\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t" Oct 30 18:19:52.721690 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:52.721657 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t" Oct 30 18:19:53.059706 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:19:53.059678 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef482c4_f1c6_4b06_8e14_220e622fa711.slice/crio-75c98c72c8bf7e74fda0c2f73c7257185428d29f15e6b52af09074c704f93793 WatchSource:0}: Error finding container 75c98c72c8bf7e74fda0c2f73c7257185428d29f15e6b52af09074c704f93793: Status 404 returned error can't find the container with id 75c98c72c8bf7e74fda0c2f73c7257185428d29f15e6b52af09074c704f93793 Oct 30 18:19:53.061638 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:53.061610 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t"] Oct 30 18:19:53.526626 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:53.526592 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t" event={"ID":"7ef482c4-f1c6-4b06-8e14-220e622fa711","Type":"ContainerStarted","Data":"4cfa2179caa134986e43e6ed7da366478a9e6e18926dd1b050bf9c8fc5e8b6df"} Oct 30 18:19:53.527077 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:53.526661 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t" event={"ID":"7ef482c4-f1c6-4b06-8e14-220e622fa711","Type":"ContainerStarted","Data":"75c98c72c8bf7e74fda0c2f73c7257185428d29f15e6b52af09074c704f93793"} Oct 30 18:19:53.527077 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:53.526808 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t" Oct 30 18:19:59.536166 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:59.536118 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t" podUID="7ef482c4-f1c6-4b06-8e14-220e622fa711" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.143:8443/health\": dial tcp 10.134.0.143:8443: connect: connection refused" Oct 30 18:19:59.965808 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:59.965762 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t" podStartSLOduration=7.636298233 podStartE2EDuration="7.965747833s" totalImagesPullingTime="329.4496ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:53.63581295 +0000 UTC m=+1646.123449985" watchObservedRunningTime="2026-10-30 18:19:59.965747833 +0000 UTC m=+1652.453384867" Oct 30 18:19:59.969948 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:59.969918 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc"] Oct 30 18:19:59.976823 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:59.976801 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc" Oct 30 18:19:59.999710 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:19:59.999662 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc"] Oct 30 18:20:00.015374 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:00.005930 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-069812-serving-cert\"" Oct 30 18:20:00.102744 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:00.102700 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ms25\" (UniqueName: \"kubernetes.io/projected/917f5f39-d463-4c4f-a122-13ed5a3d6075-kube-api-access-7ms25\") pod \"maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc\" (UID: \"917f5f39-d463-4c4f-a122-13ed5a3d6075\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc" Oct 30 18:20:00.103033 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:00.103004 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/917f5f39-d463-4c4f-a122-13ed5a3d6075-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc\" (UID: \"917f5f39-d463-4c4f-a122-13ed5a3d6075\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc" Oct 30 18:20:00.103108 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:00.103085 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/917f5f39-d463-4c4f-a122-13ed5a3d6075-maas-api-tls\") pod \"maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc\" (UID: \"917f5f39-d463-4c4f-a122-13ed5a3d6075\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc" Oct 30 18:20:00.103229 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:00.103204 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/917f5f39-d463-4c4f-a122-13ed5a3d6075-metrics-certs\") pod \"maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc\" (UID: \"917f5f39-d463-4c4f-a122-13ed5a3d6075\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc" Oct 30 18:20:00.211345 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:00.209551 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/917f5f39-d463-4c4f-a122-13ed5a3d6075-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc\" (UID: \"917f5f39-d463-4c4f-a122-13ed5a3d6075\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc" Oct 30 18:20:00.211345 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:00.209637 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/917f5f39-d463-4c4f-a122-13ed5a3d6075-maas-api-tls\") pod \"maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc\" (UID: \"917f5f39-d463-4c4f-a122-13ed5a3d6075\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc" Oct 30 18:20:00.211345 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:00.209726 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/917f5f39-d463-4c4f-a122-13ed5a3d6075-metrics-certs\") pod \"maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc\" (UID: \"917f5f39-d463-4c4f-a122-13ed5a3d6075\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc" Oct 30 18:20:00.211345 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:00.209849 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ms25\" (UniqueName: \"kubernetes.io/projected/917f5f39-d463-4c4f-a122-13ed5a3d6075-kube-api-access-7ms25\") pod \"maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc\" (UID: \"917f5f39-d463-4c4f-a122-13ed5a3d6075\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc" Oct 30 18:20:00.215355 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:00.212272 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/917f5f39-d463-4c4f-a122-13ed5a3d6075-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc\" (UID: \"917f5f39-d463-4c4f-a122-13ed5a3d6075\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc" Oct 30 18:20:00.250918 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:00.250553 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/917f5f39-d463-4c4f-a122-13ed5a3d6075-metrics-certs\") pod \"maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc\" (UID: \"917f5f39-d463-4c4f-a122-13ed5a3d6075\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc" Oct 30 18:20:00.255348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:00.251726 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ms25\" (UniqueName: \"kubernetes.io/projected/917f5f39-d463-4c4f-a122-13ed5a3d6075-kube-api-access-7ms25\") pod \"maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc\" (UID: \"917f5f39-d463-4c4f-a122-13ed5a3d6075\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc" Oct 30 18:20:00.264177 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:00.256189 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/917f5f39-d463-4c4f-a122-13ed5a3d6075-maas-api-tls\") pod \"maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc\" (UID: \"917f5f39-d463-4c4f-a122-13ed5a3d6075\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc" Oct 30 18:20:00.306159 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:00.306119 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc" Oct 30 18:20:00.502663 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:20:00.501036 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:20:00.646664 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:20:00.644440 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod917f5f39_d463_4c4f_a122_13ed5a3d6075.slice/crio-bafb78641a4b7934e5e9f65f796f4877fab16c0a908e136c14ff0f738aee23cd WatchSource:0}: Error finding container bafb78641a4b7934e5e9f65f796f4877fab16c0a908e136c14ff0f738aee23cd: Status 404 returned error can't find the container with id bafb78641a4b7934e5e9f65f796f4877fab16c0a908e136c14ff0f738aee23cd Oct 30 18:20:00.647394 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:00.647357 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc"] Oct 30 18:20:01.031536 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:01.031501 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-d9fb4d884-4b2qs"] Oct 30 18:20:01.069554 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:01.069519 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-86dcff4487-4j59n"] Oct 30 18:20:01.622545 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:01.622450 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc" event={"ID":"917f5f39-d463-4c4f-a122-13ed5a3d6075","Type":"ContainerStarted","Data":"09794edd3789b2e51595a2193f93670b875b5f38018091d107c0097a2b2c9146"} Oct 30 18:20:01.622545 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:01.622501 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc" event={"ID":"917f5f39-d463-4c4f-a122-13ed5a3d6075","Type":"ContainerStarted","Data":"bafb78641a4b7934e5e9f65f796f4877fab16c0a908e136c14ff0f738aee23cd"} Oct 30 18:20:01.623119 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:01.623090 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc" Oct 30 18:20:01.712407 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:01.706772 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc" podStartSLOduration=2.144238581 podStartE2EDuration="2.706750613s" totalImagesPullingTime="562.512032ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:01.684925287 +0000 UTC m=+1654.172562350" watchObservedRunningTime="2026-10-30 18:20:01.706750613 +0000 UTC m=+1654.194387648" Oct 30 18:20:03.812402 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:03.812183 4455 generic.go:356] "Generic (PLEG): container finished" podID="7ef482c4-f1c6-4b06-8e14-220e622fa711" containerID="4cfa2179caa134986e43e6ed7da366478a9e6e18926dd1b050bf9c8fc5e8b6df" exitCode=1 Oct 30 18:20:03.812878 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:03.812408 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t" event={"ID":"7ef482c4-f1c6-4b06-8e14-220e622fa711","Type":"ContainerDied","Data":"4cfa2179caa134986e43e6ed7da366478a9e6e18926dd1b050bf9c8fc5e8b6df"} Oct 30 18:20:03.818384 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:03.813180 4455 scope.go:122] "RemoveContainer" containerID="4cfa2179caa134986e43e6ed7da366478a9e6e18926dd1b050bf9c8fc5e8b6df" Oct 30 18:20:04.822149 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:04.822113 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t" event={"ID":"7ef482c4-f1c6-4b06-8e14-220e622fa711","Type":"ContainerStarted","Data":"225f3f9713f2f7869a356bf7a3b3756333ace26f90d011996b9a284cb85ffbf2"} Oct 30 18:20:04.822599 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:04.822316 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t" Oct 30 18:20:06.032009 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:06.031970 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-d9fb4d884-4b2qs" podUID="6cf9453e-28f8-404b-a329-64d3b3398fb6" containerName="payload-processing" containerID="cri-o://a6c037ded1088fb87560dc295febb851d5ebffa9fcbaf63af4d75cf42780618e" gracePeriod=25 Oct 30 18:20:06.070500 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:06.070445 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-86dcff4487-4j59n" podUID="2eb84843-d30a-48e9-83c6-82e5f4f6c33a" containerName="payload-pre-processing" containerID="cri-o://832514224e69eef5c3e4da5415e229621eae4f2f133cc2655919d103bc4ac375" gracePeriod=25 Oct 30 18:20:06.428976 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:06.428941 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-86dcff4487-4j59n" Oct 30 18:20:06.500262 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:06.500233 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-d9fb4d884-4b2qs" Oct 30 18:20:06.529281 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:06.529249 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eb84843-d30a-48e9-83c6-82e5f4f6c33a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb84843-d30a-48e9-83c6-82e5f4f6c33a-plugins-config-volume\") pod \"2eb84843-d30a-48e9-83c6-82e5f4f6c33a\" (UID: \"2eb84843-d30a-48e9-83c6-82e5f4f6c33a\") " Oct 30 18:20:06.529476 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:06.529387 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cf9453e-28f8-404b-a329-64d3b3398fb6-kube-api-access-c9v26\" (UniqueName: \"kubernetes.io/projected/6cf9453e-28f8-404b-a329-64d3b3398fb6-kube-api-access-c9v26\") pod \"6cf9453e-28f8-404b-a329-64d3b3398fb6\" (UID: \"6cf9453e-28f8-404b-a329-64d3b3398fb6\") " Oct 30 18:20:06.529531 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:06.529507 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eb84843-d30a-48e9-83c6-82e5f4f6c33a-kube-api-access-mxqnp\" (UniqueName: \"kubernetes.io/projected/2eb84843-d30a-48e9-83c6-82e5f4f6c33a-kube-api-access-mxqnp\") pod \"2eb84843-d30a-48e9-83c6-82e5f4f6c33a\" (UID: \"2eb84843-d30a-48e9-83c6-82e5f4f6c33a\") " Oct 30 18:20:06.529674 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:06.529650 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eb84843-d30a-48e9-83c6-82e5f4f6c33a-plugins-config-volume" pod "2eb84843-d30a-48e9-83c6-82e5f4f6c33a" (UID: "2eb84843-d30a-48e9-83c6-82e5f4f6c33a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:20:06.529763 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:06.529721 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cf9453e-28f8-404b-a329-64d3b3398fb6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cf9453e-28f8-404b-a329-64d3b3398fb6-plugins-config-volume\") pod \"6cf9453e-28f8-404b-a329-64d3b3398fb6\" (UID: \"6cf9453e-28f8-404b-a329-64d3b3398fb6\") " Oct 30 18:20:06.530063 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:06.530042 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cf9453e-28f8-404b-a329-64d3b3398fb6-plugins-config-volume" pod "6cf9453e-28f8-404b-a329-64d3b3398fb6" (UID: "6cf9453e-28f8-404b-a329-64d3b3398fb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:20:06.530341 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:06.530298 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cf9453e-28f8-404b-a329-64d3b3398fb6-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:20:06.530416 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:06.530316 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb84843-d30a-48e9-83c6-82e5f4f6c33a-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:20:06.531814 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:06.531791 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cf9453e-28f8-404b-a329-64d3b3398fb6-kube-api-access-c9v26" pod "6cf9453e-28f8-404b-a329-64d3b3398fb6" (UID: "6cf9453e-28f8-404b-a329-64d3b3398fb6"). InnerVolumeSpecName "kube-api-access-c9v26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:06.531903 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:06.531833 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eb84843-d30a-48e9-83c6-82e5f4f6c33a-kube-api-access-mxqnp" pod "2eb84843-d30a-48e9-83c6-82e5f4f6c33a" (UID: "2eb84843-d30a-48e9-83c6-82e5f4f6c33a"). InnerVolumeSpecName "kube-api-access-mxqnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:06.631890 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:06.631802 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxqnp\" (UniqueName: \"kubernetes.io/projected/2eb84843-d30a-48e9-83c6-82e5f4f6c33a-kube-api-access-mxqnp\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:20:06.631890 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:06.631840 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9v26\" (UniqueName: \"kubernetes.io/projected/6cf9453e-28f8-404b-a329-64d3b3398fb6-kube-api-access-c9v26\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:20:06.839269 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:06.839236 4455 generic.go:356] "Generic (PLEG): container finished" podID="2eb84843-d30a-48e9-83c6-82e5f4f6c33a" containerID="832514224e69eef5c3e4da5415e229621eae4f2f133cc2655919d103bc4ac375" exitCode=0 Oct 30 18:20:06.839472 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:06.839289 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-86dcff4487-4j59n" Oct 30 18:20:06.839472 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:06.839346 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-86dcff4487-4j59n" event={"ID":"2eb84843-d30a-48e9-83c6-82e5f4f6c33a","Type":"ContainerDied","Data":"832514224e69eef5c3e4da5415e229621eae4f2f133cc2655919d103bc4ac375"} Oct 30 18:20:06.839472 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:06.839404 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-86dcff4487-4j59n" event={"ID":"2eb84843-d30a-48e9-83c6-82e5f4f6c33a","Type":"ContainerDied","Data":"5d2d4d955d915eaa14dee7b4c329f4d3e2024e438bfe988ec3883d7e62b5000f"} Oct 30 18:20:06.839472 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:06.839433 4455 scope.go:122] "RemoveContainer" containerID="832514224e69eef5c3e4da5415e229621eae4f2f133cc2655919d103bc4ac375" Oct 30 18:20:06.841312 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:06.841278 4455 generic.go:356] "Generic (PLEG): container finished" podID="6cf9453e-28f8-404b-a329-64d3b3398fb6" containerID="a6c037ded1088fb87560dc295febb851d5ebffa9fcbaf63af4d75cf42780618e" exitCode=0 Oct 30 18:20:06.841509 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:06.841313 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-d9fb4d884-4b2qs" event={"ID":"6cf9453e-28f8-404b-a329-64d3b3398fb6","Type":"ContainerDied","Data":"a6c037ded1088fb87560dc295febb851d5ebffa9fcbaf63af4d75cf42780618e"} Oct 30 18:20:06.841509 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:06.841404 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-d9fb4d884-4b2qs" Oct 30 18:20:06.841509 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:06.841404 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-d9fb4d884-4b2qs" event={"ID":"6cf9453e-28f8-404b-a329-64d3b3398fb6","Type":"ContainerDied","Data":"f6777707ca37aa6e84f51bdeaa32fe6daa25fb861be2f8bdf5a3d93d1c5573ba"} Oct 30 18:20:06.857058 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:06.857037 4455 scope.go:122] "RemoveContainer" containerID="832514224e69eef5c3e4da5415e229621eae4f2f133cc2655919d103bc4ac375" Oct 30 18:20:06.857449 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:20:06.857415 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"832514224e69eef5c3e4da5415e229621eae4f2f133cc2655919d103bc4ac375\": container with ID starting with 832514224e69eef5c3e4da5415e229621eae4f2f133cc2655919d103bc4ac375 not found: ID does not exist" containerID="832514224e69eef5c3e4da5415e229621eae4f2f133cc2655919d103bc4ac375" Oct 30 18:20:06.857548 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:06.857448 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"832514224e69eef5c3e4da5415e229621eae4f2f133cc2655919d103bc4ac375"} err="failed to get container status \"832514224e69eef5c3e4da5415e229621eae4f2f133cc2655919d103bc4ac375\": rpc error: code = NotFound desc = could not find container \"832514224e69eef5c3e4da5415e229621eae4f2f133cc2655919d103bc4ac375\": container with ID starting with 832514224e69eef5c3e4da5415e229621eae4f2f133cc2655919d103bc4ac375 not found: ID does not exist" Oct 30 18:20:06.857548 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:06.857466 4455 scope.go:122] "RemoveContainer" containerID="a6c037ded1088fb87560dc295febb851d5ebffa9fcbaf63af4d75cf42780618e" Oct 30 18:20:06.868085 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:06.868063 4455 scope.go:122] "RemoveContainer" containerID="a6c037ded1088fb87560dc295febb851d5ebffa9fcbaf63af4d75cf42780618e" Oct 30 18:20:06.868402 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:20:06.868378 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6c037ded1088fb87560dc295febb851d5ebffa9fcbaf63af4d75cf42780618e\": container with ID starting with a6c037ded1088fb87560dc295febb851d5ebffa9fcbaf63af4d75cf42780618e not found: ID does not exist" containerID="a6c037ded1088fb87560dc295febb851d5ebffa9fcbaf63af4d75cf42780618e" Oct 30 18:20:06.868479 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:06.868410 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6c037ded1088fb87560dc295febb851d5ebffa9fcbaf63af4d75cf42780618e"} err="failed to get container status \"a6c037ded1088fb87560dc295febb851d5ebffa9fcbaf63af4d75cf42780618e\": rpc error: code = NotFound desc = could not find container \"a6c037ded1088fb87560dc295febb851d5ebffa9fcbaf63af4d75cf42780618e\": container with ID starting with a6c037ded1088fb87560dc295febb851d5ebffa9fcbaf63af4d75cf42780618e not found: ID does not exist" Oct 30 18:20:06.900686 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:06.900556 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-d9fb4d884-4b2qs"] Oct 30 18:20:06.926222 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:06.916383 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-d9fb4d884-4b2qs"] Oct 30 18:20:06.954138 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:06.954109 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-86dcff4487-4j59n"] Oct 30 18:20:06.958227 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:06.958204 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8dc70-86dcff4487-4j59n"] Oct 30 18:20:07.636592 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:07.636558 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eb84843-d30a-48e9-83c6-82e5f4f6c33a" path="/var/lib/kubelet/pods/2eb84843-d30a-48e9-83c6-82e5f4f6c33a/volumes" Oct 30 18:20:07.637208 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:07.637187 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cf9453e-28f8-404b-a329-64d3b3398fb6" path="/var/lib/kubelet/pods/6cf9453e-28f8-404b-a329-64d3b3398fb6/volumes" Oct 30 18:20:07.637603 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:07.637586 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc" Oct 30 18:20:09.076442 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:09.076346 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-189cda-ffc578cd-gfkwj"] Oct 30 18:20:09.115012 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:09.114976 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-fc765d467-mz9jx"] Oct 30 18:20:10.836380 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:10.836344 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t" Oct 30 18:20:14.077310 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:14.077268 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-ffc578cd-gfkwj" podUID="af2d0e78-3caa-455d-8f94-701e396a9324" containerName="payload-processing" containerID="cri-o://9793883a40a127118495dfb68ded0810b7c37fa009dbffd24893a43458692d3f" gracePeriod=25 Oct 30 18:20:14.116188 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:14.116143 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-fc765d467-mz9jx" podUID="b6b83753-e4ee-4515-af11-8c8774fee80e" containerName="payload-pre-processing" containerID="cri-o://09eb4f86b0c59f81da73fbd2cd8e010f90cbf53a8f4d8f859849ecfa2fea9c2e" gracePeriod=25 Oct 30 18:20:14.442699 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:14.442671 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-fc765d467-mz9jx" Oct 30 18:20:14.463667 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:14.463641 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-ffc578cd-gfkwj" Oct 30 18:20:14.480514 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:14.480486 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af2d0e78-3caa-455d-8f94-701e396a9324-kube-api-access-zvvhv\" (UniqueName: \"kubernetes.io/projected/af2d0e78-3caa-455d-8f94-701e396a9324-kube-api-access-zvvhv\") pod \"af2d0e78-3caa-455d-8f94-701e396a9324\" (UID: \"af2d0e78-3caa-455d-8f94-701e396a9324\") " Oct 30 18:20:14.480701 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:14.480544 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af2d0e78-3caa-455d-8f94-701e396a9324-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af2d0e78-3caa-455d-8f94-701e396a9324-plugins-config-volume\") pod \"af2d0e78-3caa-455d-8f94-701e396a9324\" (UID: \"af2d0e78-3caa-455d-8f94-701e396a9324\") " Oct 30 18:20:14.480701 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:14.480670 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b83753-e4ee-4515-af11-8c8774fee80e-kube-api-access-5qqjn\" (UniqueName: \"kubernetes.io/projected/b6b83753-e4ee-4515-af11-8c8774fee80e-kube-api-access-5qqjn\") pod \"b6b83753-e4ee-4515-af11-8c8774fee80e\" (UID: \"b6b83753-e4ee-4515-af11-8c8774fee80e\") " Oct 30 18:20:14.480807 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:14.480708 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6b83753-e4ee-4515-af11-8c8774fee80e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b83753-e4ee-4515-af11-8c8774fee80e-plugins-config-volume\") pod \"b6b83753-e4ee-4515-af11-8c8774fee80e\" (UID: \"b6b83753-e4ee-4515-af11-8c8774fee80e\") " Oct 30 18:20:14.481142 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:14.480994 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af2d0e78-3caa-455d-8f94-701e396a9324-plugins-config-volume" pod "af2d0e78-3caa-455d-8f94-701e396a9324" (UID: "af2d0e78-3caa-455d-8f94-701e396a9324"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:20:14.481310 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:14.481177 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b83753-e4ee-4515-af11-8c8774fee80e-plugins-config-volume" pod "b6b83753-e4ee-4515-af11-8c8774fee80e" (UID: "b6b83753-e4ee-4515-af11-8c8774fee80e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:20:14.481457 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:14.481434 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b83753-e4ee-4515-af11-8c8774fee80e-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:20:14.481507 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:14.481463 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af2d0e78-3caa-455d-8f94-701e396a9324-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:20:14.482989 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:14.482956 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b83753-e4ee-4515-af11-8c8774fee80e-kube-api-access-5qqjn" pod "b6b83753-e4ee-4515-af11-8c8774fee80e" (UID: "b6b83753-e4ee-4515-af11-8c8774fee80e"). InnerVolumeSpecName "kube-api-access-5qqjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:14.483176 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:14.483156 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af2d0e78-3caa-455d-8f94-701e396a9324-kube-api-access-zvvhv" pod "af2d0e78-3caa-455d-8f94-701e396a9324" (UID: "af2d0e78-3caa-455d-8f94-701e396a9324"). InnerVolumeSpecName "kube-api-access-zvvhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:14.583395 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:14.583360 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvvhv\" (UniqueName: \"kubernetes.io/projected/af2d0e78-3caa-455d-8f94-701e396a9324-kube-api-access-zvvhv\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:20:14.583395 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:14.583395 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qqjn\" (UniqueName: \"kubernetes.io/projected/b6b83753-e4ee-4515-af11-8c8774fee80e-kube-api-access-5qqjn\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:20:15.055628 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:15.055596 4455 generic.go:356] "Generic (PLEG): container finished" podID="b6b83753-e4ee-4515-af11-8c8774fee80e" containerID="09eb4f86b0c59f81da73fbd2cd8e010f90cbf53a8f4d8f859849ecfa2fea9c2e" exitCode=0 Oct 30 18:20:15.055826 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:15.055649 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-fc765d467-mz9jx" Oct 30 18:20:15.055826 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:15.055682 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-fc765d467-mz9jx" event={"ID":"b6b83753-e4ee-4515-af11-8c8774fee80e","Type":"ContainerDied","Data":"09eb4f86b0c59f81da73fbd2cd8e010f90cbf53a8f4d8f859849ecfa2fea9c2e"} Oct 30 18:20:15.055826 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:15.055739 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-fc765d467-mz9jx" event={"ID":"b6b83753-e4ee-4515-af11-8c8774fee80e","Type":"ContainerDied","Data":"1ca418a34f86cbf7e28bd3fd0482fccf1c40f40b60fe87c8eb7390229c0054c6"} Oct 30 18:20:15.055826 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:15.055767 4455 scope.go:122] "RemoveContainer" containerID="09eb4f86b0c59f81da73fbd2cd8e010f90cbf53a8f4d8f859849ecfa2fea9c2e" Oct 30 18:20:15.057502 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:15.057458 4455 generic.go:356] "Generic (PLEG): container finished" podID="af2d0e78-3caa-455d-8f94-701e396a9324" containerID="9793883a40a127118495dfb68ded0810b7c37fa009dbffd24893a43458692d3f" exitCode=0 Oct 30 18:20:15.057502 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:15.057491 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-ffc578cd-gfkwj" event={"ID":"af2d0e78-3caa-455d-8f94-701e396a9324","Type":"ContainerDied","Data":"9793883a40a127118495dfb68ded0810b7c37fa009dbffd24893a43458692d3f"} Oct 30 18:20:15.057681 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:15.057521 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-ffc578cd-gfkwj" Oct 30 18:20:15.057681 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:15.057539 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-189cda-ffc578cd-gfkwj" event={"ID":"af2d0e78-3caa-455d-8f94-701e396a9324","Type":"ContainerDied","Data":"c3d6cef86bf96a8ebfb7720a2a9aee5228009d29d8dd96f196ddc91013addd76"} Oct 30 18:20:15.068628 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:15.068512 4455 scope.go:122] "RemoveContainer" containerID="09eb4f86b0c59f81da73fbd2cd8e010f90cbf53a8f4d8f859849ecfa2fea9c2e" Oct 30 18:20:15.069312 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:20:15.068855 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09eb4f86b0c59f81da73fbd2cd8e010f90cbf53a8f4d8f859849ecfa2fea9c2e\": container with ID starting with 09eb4f86b0c59f81da73fbd2cd8e010f90cbf53a8f4d8f859849ecfa2fea9c2e not found: ID does not exist" containerID="09eb4f86b0c59f81da73fbd2cd8e010f90cbf53a8f4d8f859849ecfa2fea9c2e" Oct 30 18:20:15.069312 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:15.068891 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09eb4f86b0c59f81da73fbd2cd8e010f90cbf53a8f4d8f859849ecfa2fea9c2e"} err="failed to get container status \"09eb4f86b0c59f81da73fbd2cd8e010f90cbf53a8f4d8f859849ecfa2fea9c2e\": rpc error: code = NotFound desc = could not find container \"09eb4f86b0c59f81da73fbd2cd8e010f90cbf53a8f4d8f859849ecfa2fea9c2e\": container with ID starting with 09eb4f86b0c59f81da73fbd2cd8e010f90cbf53a8f4d8f859849ecfa2fea9c2e not found: ID does not exist" Oct 30 18:20:15.069312 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:15.068919 4455 scope.go:122] "RemoveContainer" containerID="9793883a40a127118495dfb68ded0810b7c37fa009dbffd24893a43458692d3f" Oct 30 18:20:15.079846 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:15.079681 4455 scope.go:122] "RemoveContainer" containerID="9793883a40a127118495dfb68ded0810b7c37fa009dbffd24893a43458692d3f" Oct 30 18:20:15.080108 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:20:15.079987 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9793883a40a127118495dfb68ded0810b7c37fa009dbffd24893a43458692d3f\": container with ID starting with 9793883a40a127118495dfb68ded0810b7c37fa009dbffd24893a43458692d3f not found: ID does not exist" containerID="9793883a40a127118495dfb68ded0810b7c37fa009dbffd24893a43458692d3f" Oct 30 18:20:15.080108 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:15.080025 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9793883a40a127118495dfb68ded0810b7c37fa009dbffd24893a43458692d3f"} err="failed to get container status \"9793883a40a127118495dfb68ded0810b7c37fa009dbffd24893a43458692d3f\": rpc error: code = NotFound desc = could not find container \"9793883a40a127118495dfb68ded0810b7c37fa009dbffd24893a43458692d3f\": container with ID starting with 9793883a40a127118495dfb68ded0810b7c37fa009dbffd24893a43458692d3f not found: ID does not exist" Oct 30 18:20:15.165159 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:15.165118 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-189cda-ffc578cd-gfkwj"] Oct 30 18:20:15.166009 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:15.165980 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-189cda-ffc578cd-gfkwj"] Oct 30 18:20:15.241347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:15.236381 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-fc765d467-mz9jx"] Oct 30 18:20:15.331424 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:15.322174 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-189cda-fc765d467-mz9jx"] Oct 30 18:20:15.637438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:15.637364 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af2d0e78-3caa-455d-8f94-701e396a9324" path="/var/lib/kubelet/pods/af2d0e78-3caa-455d-8f94-701e396a9324/volumes" Oct 30 18:20:15.637797 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:15.637784 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6b83753-e4ee-4515-af11-8c8774fee80e" path="/var/lib/kubelet/pods/b6b83753-e4ee-4515-af11-8c8774fee80e/volumes" Oct 30 18:20:20.472393 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:20.471871 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:20:22.497352 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:22.470706 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:20:23.106182 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:23.100443 4455 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:20:23.106182 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:23.100531 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" podUID="bbf4bc50-3670-40de-8159-3d8805fdd601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:23.607512 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:23.604929 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:23.607512 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:23.605281 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:20:24.612925 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:24.610900 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:20:25.660345 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:25.659447 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:26.264476 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:26.264441 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:20:30.397780 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:20:30.397737 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:20:31.181726 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:31.181693 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-bfb464d74-l4lmt"] Oct 30 18:20:31.182485 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:31.182470 4455 cpu_manager.go:414] "containerMap: removing container" podUID="2eb84843-d30a-48e9-83c6-82e5f4f6c33a" containerName="payload-pre-processing" Oct 30 18:20:31.182560 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:31.182486 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eb84843-d30a-48e9-83c6-82e5f4f6c33a" containerName="payload-pre-processing" Oct 30 18:20:31.182560 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:31.182504 4455 cpu_manager.go:414] "containerMap: removing container" podUID="b6b83753-e4ee-4515-af11-8c8774fee80e" containerName="payload-pre-processing" Oct 30 18:20:31.182560 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:31.182510 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6b83753-e4ee-4515-af11-8c8774fee80e" containerName="payload-pre-processing" Oct 30 18:20:31.182560 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:31.182522 4455 cpu_manager.go:414] "containerMap: removing container" podUID="af2d0e78-3caa-455d-8f94-701e396a9324" containerName="payload-processing" Oct 30 18:20:31.182560 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:31.182526 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af2d0e78-3caa-455d-8f94-701e396a9324" containerName="payload-processing" Oct 30 18:20:31.182560 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:31.182546 4455 cpu_manager.go:414] "containerMap: removing container" podUID="6cf9453e-28f8-404b-a329-64d3b3398fb6" containerName="payload-processing" Oct 30 18:20:31.182560 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:31.182550 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cf9453e-28f8-404b-a329-64d3b3398fb6" containerName="payload-processing" Oct 30 18:20:31.182779 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:31.182667 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eb84843-d30a-48e9-83c6-82e5f4f6c33a" containerName="payload-pre-processing" Oct 30 18:20:31.182779 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:31.182678 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6b83753-e4ee-4515-af11-8c8774fee80e" containerName="payload-pre-processing" Oct 30 18:20:31.182779 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:31.182687 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="af2d0e78-3caa-455d-8f94-701e396a9324" containerName="payload-processing" Oct 30 18:20:31.182779 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:31.182695 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cf9453e-28f8-404b-a329-64d3b3398fb6" containerName="payload-processing" Oct 30 18:20:31.189816 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:31.189788 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-bfb464d74-l4lmt" Oct 30 18:20:31.212397 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:31.212367 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-bfb464d74-l4lmt"] Oct 30 18:20:31.225042 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:31.225011 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a8dc70\"" Oct 30 18:20:31.225587 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:31.225562 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a8dc70-dockercfg-ln2qg\"" Oct 30 18:20:31.355366 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:31.349291 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e55a9f3-53be-4afd-be10-c7a144bc9835-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a8dc70-bfb464d74-l4lmt\" (UID: \"8e55a9f3-53be-4afd-be10-c7a144bc9835\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-bfb464d74-l4lmt" Oct 30 18:20:31.355366 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:31.349421 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f4rf\" (UniqueName: \"kubernetes.io/projected/8e55a9f3-53be-4afd-be10-c7a144bc9835-kube-api-access-5f4rf\") pod \"payload-processing-e2e-ipp-b-a8dc70-bfb464d74-l4lmt\" (UID: \"8e55a9f3-53be-4afd-be10-c7a144bc9835\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-bfb464d74-l4lmt" Oct 30 18:20:31.451039 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:31.450945 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e55a9f3-53be-4afd-be10-c7a144bc9835-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a8dc70-bfb464d74-l4lmt\" (UID: \"8e55a9f3-53be-4afd-be10-c7a144bc9835\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-bfb464d74-l4lmt" Oct 30 18:20:31.451039 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:31.451003 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f4rf\" (UniqueName: \"kubernetes.io/projected/8e55a9f3-53be-4afd-be10-c7a144bc9835-kube-api-access-5f4rf\") pod \"payload-processing-e2e-ipp-b-a8dc70-bfb464d74-l4lmt\" (UID: \"8e55a9f3-53be-4afd-be10-c7a144bc9835\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-bfb464d74-l4lmt" Oct 30 18:20:31.451856 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:31.451834 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e55a9f3-53be-4afd-be10-c7a144bc9835-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a8dc70-bfb464d74-l4lmt\" (UID: \"8e55a9f3-53be-4afd-be10-c7a144bc9835\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-bfb464d74-l4lmt" Oct 30 18:20:31.477056 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:31.477018 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f4rf\" (UniqueName: \"kubernetes.io/projected/8e55a9f3-53be-4afd-be10-c7a144bc9835-kube-api-access-5f4rf\") pod \"payload-processing-e2e-ipp-b-a8dc70-bfb464d74-l4lmt\" (UID: \"8e55a9f3-53be-4afd-be10-c7a144bc9835\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-bfb464d74-l4lmt" Oct 30 18:20:31.512692 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:31.512655 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-bfb464d74-l4lmt" Oct 30 18:20:31.818667 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:20:31.818626 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e55a9f3_53be_4afd_be10_c7a144bc9835.slice/crio-45ca875973d45dd945e4cfc6f932971eefd1123d5610c77a338abd0587aed0da WatchSource:0}: Error finding container 45ca875973d45dd945e4cfc6f932971eefd1123d5610c77a338abd0587aed0da: Status 404 returned error can't find the container with id 45ca875973d45dd945e4cfc6f932971eefd1123d5610c77a338abd0587aed0da Oct 30 18:20:31.821293 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:31.821268 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-bfb464d74-l4lmt"] Oct 30 18:20:32.825695 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:32.825657 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-bfb464d74-l4lmt" event={"ID":"8e55a9f3-53be-4afd-be10-c7a144bc9835","Type":"ContainerStarted","Data":"65ced1793e29a03c9042e3907866d410a35ed38652f2e10f38c21b12a99aa833"} Oct 30 18:20:32.826138 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:32.825723 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-bfb464d74-l4lmt" event={"ID":"8e55a9f3-53be-4afd-be10-c7a144bc9835","Type":"ContainerStarted","Data":"45ca875973d45dd945e4cfc6f932971eefd1123d5610c77a338abd0587aed0da"} Oct 30 18:20:32.826138 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:32.825762 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-bfb464d74-l4lmt" Oct 30 18:20:32.875733 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:32.875666 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-bfb464d74-l4lmt" podStartSLOduration=1.87565129 podStartE2EDuration="1.87565129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:32.871826144 +0000 UTC m=+1685.359463179" watchObservedRunningTime="2026-10-30 18:20:32.87565129 +0000 UTC m=+1685.363288368" Oct 30 18:20:35.977357 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:35.977301 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz"] Oct 30 18:20:35.982983 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:35.982957 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz" Oct 30 18:20:36.005043 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:36.005007 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz"] Oct 30 18:20:36.005248 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:36.005235 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-394a31-serving-cert\"" Oct 30 18:20:36.035483 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:36.035445 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/466c7c11-2221-4749-bba1-d0f2383c66d2-metrics-certs\") pod \"maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz\" (UID: \"466c7c11-2221-4749-bba1-d0f2383c66d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz" Oct 30 18:20:36.035915 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:36.035889 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjl8n\" (UniqueName: \"kubernetes.io/projected/466c7c11-2221-4749-bba1-d0f2383c66d2-kube-api-access-fjl8n\") pod \"maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz\" (UID: \"466c7c11-2221-4749-bba1-d0f2383c66d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz" Oct 30 18:20:36.036218 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:36.036197 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/466c7c11-2221-4749-bba1-d0f2383c66d2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz\" (UID: \"466c7c11-2221-4749-bba1-d0f2383c66d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz" Oct 30 18:20:36.036457 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:36.036435 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/466c7c11-2221-4749-bba1-d0f2383c66d2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz\" (UID: \"466c7c11-2221-4749-bba1-d0f2383c66d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz" Oct 30 18:20:36.138290 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:36.138259 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/466c7c11-2221-4749-bba1-d0f2383c66d2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz\" (UID: \"466c7c11-2221-4749-bba1-d0f2383c66d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz" Oct 30 18:20:36.138488 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:36.138408 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/466c7c11-2221-4749-bba1-d0f2383c66d2-metrics-certs\") pod \"maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz\" (UID: \"466c7c11-2221-4749-bba1-d0f2383c66d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz" Oct 30 18:20:36.138488 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:36.138479 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjl8n\" (UniqueName: \"kubernetes.io/projected/466c7c11-2221-4749-bba1-d0f2383c66d2-kube-api-access-fjl8n\") pod \"maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz\" (UID: \"466c7c11-2221-4749-bba1-d0f2383c66d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz" Oct 30 18:20:36.138630 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:36.138613 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/466c7c11-2221-4749-bba1-d0f2383c66d2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz\" (UID: \"466c7c11-2221-4749-bba1-d0f2383c66d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz" Oct 30 18:20:36.138800 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:20:36.138782 4455 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-394a31-serving-cert: secret "maas-api-e2e-ipp-b-394a31-serving-cert" not found Oct 30 18:20:36.138913 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:20:36.138901 4455 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/466c7c11-2221-4749-bba1-d0f2383c66d2-maas-api-tls podName:466c7c11-2221-4749-bba1-d0f2383c66d2 nodeName:}" failed. No retries permitted until 2026-10-30 18:20:36.638870838 +0000 UTC m=+1689.126507849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/466c7c11-2221-4749-bba1-d0f2383c66d2-maas-api-tls") pod "maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz" (UID: "466c7c11-2221-4749-bba1-d0f2383c66d2") : secret "maas-api-e2e-ipp-b-394a31-serving-cert" not found Oct 30 18:20:36.139087 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:36.139059 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/466c7c11-2221-4749-bba1-d0f2383c66d2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz\" (UID: \"466c7c11-2221-4749-bba1-d0f2383c66d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz" Oct 30 18:20:36.141425 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:36.141397 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/466c7c11-2221-4749-bba1-d0f2383c66d2-metrics-certs\") pod \"maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz\" (UID: \"466c7c11-2221-4749-bba1-d0f2383c66d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz" Oct 30 18:20:36.164705 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:36.164682 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjl8n\" (UniqueName: \"kubernetes.io/projected/466c7c11-2221-4749-bba1-d0f2383c66d2-kube-api-access-fjl8n\") pod \"maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz\" (UID: \"466c7c11-2221-4749-bba1-d0f2383c66d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz" Oct 30 18:20:36.652621 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:36.652587 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/466c7c11-2221-4749-bba1-d0f2383c66d2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz\" (UID: \"466c7c11-2221-4749-bba1-d0f2383c66d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz" Oct 30 18:20:36.655581 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:36.655544 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/466c7c11-2221-4749-bba1-d0f2383c66d2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz\" (UID: \"466c7c11-2221-4749-bba1-d0f2383c66d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz" Oct 30 18:20:36.915402 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:36.911639 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz" Oct 30 18:20:37.390694 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:37.390661 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz"] Oct 30 18:20:37.943635 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:37.943602 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz" event={"ID":"466c7c11-2221-4749-bba1-d0f2383c66d2","Type":"ContainerStarted","Data":"65d681be6e180e4267af0626e72a02dff490c0a61f7b59322c3db7cc52bfb00d"} Oct 30 18:20:37.943811 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:37.943657 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz" event={"ID":"466c7c11-2221-4749-bba1-d0f2383c66d2","Type":"ContainerStarted","Data":"cb51243ddb5b149a36e5979b772b7bdc4b9e32bf958c9cabed0333cc3dbff3a6"} Oct 30 18:20:37.943811 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:37.943687 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz" Oct 30 18:20:37.991827 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:37.991763 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz" podStartSLOduration=2.608991685 podStartE2EDuration="2.99174435s" totalImagesPullingTime="382.752665ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:37.9881769 +0000 UTC m=+1690.475813936" watchObservedRunningTime="2026-10-30 18:20:37.99174435 +0000 UTC m=+1690.479381385" Oct 30 18:20:39.251214 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:39.251162 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk"] Oct 30 18:20:39.259300 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:39.259274 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" Oct 30 18:20:39.270649 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:39.269948 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk"] Oct 30 18:20:39.304984 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:39.304942 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc3ad56-5026-4312-85fb-0f40baceeee9-credential-socket\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" Oct 30 18:20:39.305269 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:39.305249 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bc3ad56-5026-4312-85fb-0f40baceeee9-istiod-ca-cert\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" Oct 30 18:20:39.305513 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:39.305495 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v9hz\" (UniqueName: \"kubernetes.io/projected/2bc3ad56-5026-4312-85fb-0f40baceeee9-kube-api-access-2v9hz\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" Oct 30 18:20:39.305718 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:39.305704 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bc3ad56-5026-4312-85fb-0f40baceeee9-istio-podinfo\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" Oct 30 18:20:39.305891 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:39.305877 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bc3ad56-5026-4312-85fb-0f40baceeee9-istio-token\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" Oct 30 18:20:39.306021 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:39.305999 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bc3ad56-5026-4312-85fb-0f40baceeee9-workload-certs\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" Oct 30 18:20:39.306212 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:39.306156 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bc3ad56-5026-4312-85fb-0f40baceeee9-istio-envoy\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" Oct 30 18:20:39.308445 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:39.308419 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bc3ad56-5026-4312-85fb-0f40baceeee9-istio-data\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" Oct 30 18:20:39.309581 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:39.308624 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc3ad56-5026-4312-85fb-0f40baceeee9-workload-socket\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" Oct 30 18:20:39.341874 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:39.341841 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-189cda-openshift-default-dockercfg-kv2k4\"" Oct 30 18:20:39.411229 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:39.411184 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v9hz\" (UniqueName: \"kubernetes.io/projected/2bc3ad56-5026-4312-85fb-0f40baceeee9-kube-api-access-2v9hz\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" Oct 30 18:20:39.411427 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:39.411262 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bc3ad56-5026-4312-85fb-0f40baceeee9-istio-podinfo\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" Oct 30 18:20:39.411427 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:39.411309 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bc3ad56-5026-4312-85fb-0f40baceeee9-istio-token\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" Oct 30 18:20:39.411427 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:39.411352 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bc3ad56-5026-4312-85fb-0f40baceeee9-workload-certs\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" Oct 30 18:20:39.411427 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:39.411398 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bc3ad56-5026-4312-85fb-0f40baceeee9-istio-envoy\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" Oct 30 18:20:39.411678 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:39.411463 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bc3ad56-5026-4312-85fb-0f40baceeee9-istio-data\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" Oct 30 18:20:39.411678 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:39.411499 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc3ad56-5026-4312-85fb-0f40baceeee9-workload-socket\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" Oct 30 18:20:39.411678 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:39.411670 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc3ad56-5026-4312-85fb-0f40baceeee9-credential-socket\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" Oct 30 18:20:39.411841 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:39.411741 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bc3ad56-5026-4312-85fb-0f40baceeee9-istiod-ca-cert\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" Oct 30 18:20:39.411998 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:39.411973 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bc3ad56-5026-4312-85fb-0f40baceeee9-istio-data\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" Oct 30 18:20:39.412063 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:39.412046 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc3ad56-5026-4312-85fb-0f40baceeee9-workload-socket\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" Oct 30 18:20:39.412234 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:39.412211 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc3ad56-5026-4312-85fb-0f40baceeee9-credential-socket\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" Oct 30 18:20:39.412355 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:39.412213 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bc3ad56-5026-4312-85fb-0f40baceeee9-workload-certs\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" Oct 30 18:20:39.412538 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:39.412513 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bc3ad56-5026-4312-85fb-0f40baceeee9-istiod-ca-cert\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" Oct 30 18:20:39.414670 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:39.414566 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bc3ad56-5026-4312-85fb-0f40baceeee9-istio-podinfo\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" Oct 30 18:20:39.414670 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:39.414619 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bc3ad56-5026-4312-85fb-0f40baceeee9-istio-envoy\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" Oct 30 18:20:39.438630 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:39.438600 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bc3ad56-5026-4312-85fb-0f40baceeee9-istio-token\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" Oct 30 18:20:39.438770 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:39.438723 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v9hz\" (UniqueName: \"kubernetes.io/projected/2bc3ad56-5026-4312-85fb-0f40baceeee9-kube-api-access-2v9hz\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" Oct 30 18:20:39.601236 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:39.595250 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" Oct 30 18:20:40.107670 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:40.104766 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk"] Oct 30 18:20:40.107670 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:20:40.104899 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc3ad56_5026_4312_85fb_0f40baceeee9.slice/crio-5f0b02c08442f628f08da7608f824aba2a64d39f2edba41d810bd23a16cea7f4 WatchSource:0}: Error finding container 5f0b02c08442f628f08da7608f824aba2a64d39f2edba41d810bd23a16cea7f4: Status 404 returned error can't find the container with id 5f0b02c08442f628f08da7608f824aba2a64d39f2edba41d810bd23a16cea7f4 Oct 30 18:20:40.112124 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:40.112081 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:20:40.112769 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:40.112741 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:20:40.112942 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:40.112923 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:20:41.009892 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:41.009853 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" event={"ID":"2bc3ad56-5026-4312-85fb-0f40baceeee9","Type":"ContainerStarted","Data":"7673c400619052abf4ce86bec66307cfc7501390202d5aca35f03e2900ce264c"} Oct 30 18:20:41.010445 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:41.009911 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" event={"ID":"2bc3ad56-5026-4312-85fb-0f40baceeee9","Type":"ContainerStarted","Data":"5f0b02c08442f628f08da7608f824aba2a64d39f2edba41d810bd23a16cea7f4"} Oct 30 18:20:41.122568 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:41.118890 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" podStartSLOduration=2.118869703 podStartE2EDuration="2.118869703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:41.090644972 +0000 UTC m=+1693.578282007" watchObservedRunningTime="2026-10-30 18:20:41.118869703 +0000 UTC m=+1693.606506754" Oct 30 18:20:41.600379 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:41.600344 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" Oct 30 18:20:41.600596 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:41.600513 4455 patch_prober.go:29] interesting pod/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 18:20:41.600596 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:41.600580 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" podUID="2bc3ad56-5026-4312-85fb-0f40baceeee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 18:20:42.607347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:42.607138 4455 patch_prober.go:29] interesting pod/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 18:20:42.607347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:42.607224 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" podUID="2bc3ad56-5026-4312-85fb-0f40baceeee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 18:20:43.597350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:43.596748 4455 patch_prober.go:29] interesting pod/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 18:20:43.597350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:43.596815 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" podUID="2bc3ad56-5026-4312-85fb-0f40baceeee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 18:20:43.855110 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:43.854514 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-bfb464d74-l4lmt" Oct 30 18:20:43.988346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:43.974298 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz" Oct 30 18:20:44.599861 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:44.599823 4455 patch_prober.go:29] interesting pod/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 18:20:44.600093 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:44.599891 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" podUID="2bc3ad56-5026-4312-85fb-0f40baceeee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 18:20:45.597825 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:45.597788 4455 patch_prober.go:29] interesting pod/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 18:20:45.598434 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:45.598406 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" podUID="2bc3ad56-5026-4312-85fb-0f40baceeee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 18:20:46.599086 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:46.599050 4455 patch_prober.go:29] interesting pod/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 18:20:46.599734 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:46.599698 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" podUID="2bc3ad56-5026-4312-85fb-0f40baceeee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 18:20:47.602841 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:47.602808 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" Oct 30 18:20:47.603281 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:47.603246 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" Oct 30 18:20:47.604826 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:20:47.604795 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" Oct 30 18:21:00.388299 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:21:00.388255 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:21:02.964452 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:02.964417 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z"] Oct 30 18:21:02.965202 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:02.965165 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z" podUID="916bd30a-6453-477b-9675-8356aea0eaba" containerName="maas-api" containerID="cri-o://9242162d8e5593f821b02df1b897a5a4885928c5d63a11b487902ef5262de0a4" gracePeriod=30 Oct 30 18:21:03.115043 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:03.114992 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-064408-c74cbb9f4-sct25"] Oct 30 18:21:03.115545 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:03.115515 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-c74cbb9f4-sct25" podUID="c3383bd2-d483-4239-8b12-879f8508d63a" containerName="payload-processing" containerID="cri-o://776a79f202bd45fad1c6c9aa7100a1a2ca71518d8471451c86047b86693d9fdc" gracePeriod=30 Oct 30 18:21:03.273287 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:03.273251 4455 generic.go:356] "Generic (PLEG): container finished" podID="916bd30a-6453-477b-9675-8356aea0eaba" containerID="9242162d8e5593f821b02df1b897a5a4885928c5d63a11b487902ef5262de0a4" exitCode=0 Oct 30 18:21:03.273679 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:03.273621 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z" event={"ID":"916bd30a-6453-477b-9675-8356aea0eaba","Type":"ContainerDied","Data":"9242162d8e5593f821b02df1b897a5a4885928c5d63a11b487902ef5262de0a4"} Oct 30 18:21:03.276874 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:03.276851 4455 generic.go:356] "Generic (PLEG): container finished" podID="c3383bd2-d483-4239-8b12-879f8508d63a" containerID="776a79f202bd45fad1c6c9aa7100a1a2ca71518d8471451c86047b86693d9fdc" exitCode=0 Oct 30 18:21:03.277178 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:03.277162 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-c74cbb9f4-sct25" event={"ID":"c3383bd2-d483-4239-8b12-879f8508d63a","Type":"ContainerDied","Data":"776a79f202bd45fad1c6c9aa7100a1a2ca71518d8471451c86047b86693d9fdc"} Oct 30 18:21:03.344902 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:03.344873 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z" Oct 30 18:21:03.457349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:03.451702 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/916bd30a-6453-477b-9675-8356aea0eaba-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/916bd30a-6453-477b-9675-8356aea0eaba-maas-api-tls\") pod \"916bd30a-6453-477b-9675-8356aea0eaba\" (UID: \"916bd30a-6453-477b-9675-8356aea0eaba\") " Oct 30 18:21:03.457349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:03.451794 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/916bd30a-6453-477b-9675-8356aea0eaba-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/916bd30a-6453-477b-9675-8356aea0eaba-openshift-service-ca\") pod \"916bd30a-6453-477b-9675-8356aea0eaba\" (UID: \"916bd30a-6453-477b-9675-8356aea0eaba\") " Oct 30 18:21:03.457349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:03.451837 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/916bd30a-6453-477b-9675-8356aea0eaba-metrics-certs\" (UniqueName: \"kubernetes.io/secret/916bd30a-6453-477b-9675-8356aea0eaba-metrics-certs\") pod \"916bd30a-6453-477b-9675-8356aea0eaba\" (UID: \"916bd30a-6453-477b-9675-8356aea0eaba\") " Oct 30 18:21:03.457349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:03.451902 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/916bd30a-6453-477b-9675-8356aea0eaba-kube-api-access-h5d6t\" (UniqueName: \"kubernetes.io/projected/916bd30a-6453-477b-9675-8356aea0eaba-kube-api-access-h5d6t\") pod \"916bd30a-6453-477b-9675-8356aea0eaba\" (UID: \"916bd30a-6453-477b-9675-8356aea0eaba\") " Oct 30 18:21:03.578949 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:03.556972 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/916bd30a-6453-477b-9675-8356aea0eaba-metrics-certs" pod "916bd30a-6453-477b-9675-8356aea0eaba" (UID: "916bd30a-6453-477b-9675-8356aea0eaba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:21:03.578949 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:03.562594 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/916bd30a-6453-477b-9675-8356aea0eaba-openshift-service-ca" pod "916bd30a-6453-477b-9675-8356aea0eaba" (UID: "916bd30a-6453-477b-9675-8356aea0eaba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:21:03.592899 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:03.591676 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/916bd30a-6453-477b-9675-8356aea0eaba-maas-api-tls" pod "916bd30a-6453-477b-9675-8356aea0eaba" (UID: "916bd30a-6453-477b-9675-8356aea0eaba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:21:03.592899 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:03.591981 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916bd30a-6453-477b-9675-8356aea0eaba-kube-api-access-h5d6t" pod "916bd30a-6453-477b-9675-8356aea0eaba" (UID: "916bd30a-6453-477b-9675-8356aea0eaba"). InnerVolumeSpecName "kube-api-access-h5d6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:03.631943 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:03.625478 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/916bd30a-6453-477b-9675-8356aea0eaba-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:21:03.631943 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:03.625517 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/916bd30a-6453-477b-9675-8356aea0eaba-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:21:03.631943 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:03.625534 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/916bd30a-6453-477b-9675-8356aea0eaba-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:21:03.631943 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:03.625550 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5d6t\" (UniqueName: \"kubernetes.io/projected/916bd30a-6453-477b-9675-8356aea0eaba-kube-api-access-h5d6t\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:21:03.752415 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:03.752390 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-c74cbb9f4-sct25" Oct 30 18:21:03.833061 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:03.831881 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3383bd2-d483-4239-8b12-879f8508d63a-kube-api-access-klh6g\" (UniqueName: \"kubernetes.io/projected/c3383bd2-d483-4239-8b12-879f8508d63a-kube-api-access-klh6g\") pod \"c3383bd2-d483-4239-8b12-879f8508d63a\" (UID: \"c3383bd2-d483-4239-8b12-879f8508d63a\") " Oct 30 18:21:03.833061 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:03.832168 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3383bd2-d483-4239-8b12-879f8508d63a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3383bd2-d483-4239-8b12-879f8508d63a-plugins-config-volume\") pod \"c3383bd2-d483-4239-8b12-879f8508d63a\" (UID: \"c3383bd2-d483-4239-8b12-879f8508d63a\") " Oct 30 18:21:03.833858 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:03.833811 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3383bd2-d483-4239-8b12-879f8508d63a-plugins-config-volume" pod "c3383bd2-d483-4239-8b12-879f8508d63a" (UID: "c3383bd2-d483-4239-8b12-879f8508d63a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:21:03.838111 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:03.837625 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3383bd2-d483-4239-8b12-879f8508d63a-kube-api-access-klh6g" pod "c3383bd2-d483-4239-8b12-879f8508d63a" (UID: "c3383bd2-d483-4239-8b12-879f8508d63a"). InnerVolumeSpecName "kube-api-access-klh6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:03.935055 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:03.934971 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klh6g\" (UniqueName: \"kubernetes.io/projected/c3383bd2-d483-4239-8b12-879f8508d63a-kube-api-access-klh6g\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:21:03.935055 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:03.935051 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3383bd2-d483-4239-8b12-879f8508d63a-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:21:04.287025 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:04.286994 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z" Oct 30 18:21:04.287453 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:04.287028 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z" event={"ID":"916bd30a-6453-477b-9675-8356aea0eaba","Type":"ContainerDied","Data":"1170bcfbaad01d5e453ad3baec7cafdbd07864fb343708e7f3a109d18bd50460"} Oct 30 18:21:04.287453 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:04.287096 4455 scope.go:122] "RemoveContainer" containerID="9242162d8e5593f821b02df1b897a5a4885928c5d63a11b487902ef5262de0a4" Oct 30 18:21:04.290526 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:04.289451 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-c74cbb9f4-sct25" Oct 30 18:21:04.290526 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:04.289307 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-064408-c74cbb9f4-sct25" event={"ID":"c3383bd2-d483-4239-8b12-879f8508d63a","Type":"ContainerDied","Data":"24089a2bc31cd65a047972c0afb9baae0cf575a77d248fe42fb195cd9ab01aca"} Oct 30 18:21:04.300637 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:04.300615 4455 scope.go:122] "RemoveContainer" containerID="776a79f202bd45fad1c6c9aa7100a1a2ca71518d8471451c86047b86693d9fdc" Oct 30 18:21:04.345786 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:04.345752 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-064408-c74cbb9f4-sct25"] Oct 30 18:21:04.358426 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:04.358396 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-064408-c74cbb9f4-sct25"] Oct 30 18:21:04.392785 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:04.392751 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z"] Oct 30 18:21:04.399287 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:04.399259 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-064408-7c5645bfbf-rps2z"] Oct 30 18:21:05.794351 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:05.793753 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="916bd30a-6453-477b-9675-8356aea0eaba" path="/var/lib/kubelet/pods/916bd30a-6453-477b-9675-8356aea0eaba/volumes" Oct 30 18:21:05.813229 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:05.813179 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3383bd2-d483-4239-8b12-879f8508d63a" path="/var/lib/kubelet/pods/c3383bd2-d483-4239-8b12-879f8508d63a/volumes" Oct 30 18:21:05.815076 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:05.815053 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-5fc46c9879-vzjtw"] Oct 30 18:21:05.828332 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:05.828274 4455 cpu_manager.go:414] "containerMap: removing container" podUID="916bd30a-6453-477b-9675-8356aea0eaba" containerName="maas-api" Oct 30 18:21:05.828332 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:05.828312 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="916bd30a-6453-477b-9675-8356aea0eaba" containerName="maas-api" Oct 30 18:21:05.828839 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:05.828441 4455 cpu_manager.go:414] "containerMap: removing container" podUID="c3383bd2-d483-4239-8b12-879f8508d63a" containerName="payload-processing" Oct 30 18:21:05.828839 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:05.828451 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3383bd2-d483-4239-8b12-879f8508d63a" containerName="payload-processing" Oct 30 18:21:05.828986 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:05.828887 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="916bd30a-6453-477b-9675-8356aea0eaba" containerName="maas-api" Oct 30 18:21:05.828986 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:05.828931 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3383bd2-d483-4239-8b12-879f8508d63a" containerName="payload-processing" Oct 30 18:21:05.846351 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:05.845076 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-5fc46c9879-vzjtw"] Oct 30 18:21:05.846351 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:05.845607 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-5fc46c9879-vzjtw" Oct 30 18:21:05.869400 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:05.868763 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-069812\"" Oct 30 18:21:05.887235 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:05.882866 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-069812-dockercfg-9t6ft\"" Oct 30 18:21:06.005752 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:06.005706 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvjjn\" (UniqueName: \"kubernetes.io/projected/cbcc3d28-dd08-4146-91d5-9138a5057230-kube-api-access-fvjjn\") pod \"payload-pre-processing-e2e-ipp-b-069812-5fc46c9879-vzjtw\" (UID: \"cbcc3d28-dd08-4146-91d5-9138a5057230\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-5fc46c9879-vzjtw" Oct 30 18:21:06.005975 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:06.005878 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbcc3d28-dd08-4146-91d5-9138a5057230-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-069812-5fc46c9879-vzjtw\" (UID: \"cbcc3d28-dd08-4146-91d5-9138a5057230\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-5fc46c9879-vzjtw" Oct 30 18:21:06.115476 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:06.114373 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbcc3d28-dd08-4146-91d5-9138a5057230-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-069812-5fc46c9879-vzjtw\" (UID: \"cbcc3d28-dd08-4146-91d5-9138a5057230\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-5fc46c9879-vzjtw" Oct 30 18:21:06.115476 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:06.114705 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvjjn\" (UniqueName: \"kubernetes.io/projected/cbcc3d28-dd08-4146-91d5-9138a5057230-kube-api-access-fvjjn\") pod \"payload-pre-processing-e2e-ipp-b-069812-5fc46c9879-vzjtw\" (UID: \"cbcc3d28-dd08-4146-91d5-9138a5057230\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-5fc46c9879-vzjtw" Oct 30 18:21:06.116056 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:06.116024 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbcc3d28-dd08-4146-91d5-9138a5057230-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-069812-5fc46c9879-vzjtw\" (UID: \"cbcc3d28-dd08-4146-91d5-9138a5057230\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-5fc46c9879-vzjtw" Oct 30 18:21:06.143301 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:06.143260 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvjjn\" (UniqueName: \"kubernetes.io/projected/cbcc3d28-dd08-4146-91d5-9138a5057230-kube-api-access-fvjjn\") pod \"payload-pre-processing-e2e-ipp-b-069812-5fc46c9879-vzjtw\" (UID: \"cbcc3d28-dd08-4146-91d5-9138a5057230\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-5fc46c9879-vzjtw" Oct 30 18:21:06.188059 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:06.188014 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-5fc46c9879-vzjtw" Oct 30 18:21:06.531784 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:21:06.531718 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbcc3d28_dd08_4146_91d5_9138a5057230.slice/crio-58dd9c55b7ac1c219c0d0e5ec45a6a2cdc6ecc31cd9be04b417893dda1441571 WatchSource:0}: Error finding container 58dd9c55b7ac1c219c0d0e5ec45a6a2cdc6ecc31cd9be04b417893dda1441571: Status 404 returned error can't find the container with id 58dd9c55b7ac1c219c0d0e5ec45a6a2cdc6ecc31cd9be04b417893dda1441571 Oct 30 18:21:06.538583 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:06.538548 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-5fc46c9879-vzjtw"] Oct 30 18:21:07.323428 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:07.323386 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-5fc46c9879-vzjtw" event={"ID":"cbcc3d28-dd08-4146-91d5-9138a5057230","Type":"ContainerStarted","Data":"4785b788ffa6b3b7f0e2721d0503e49445134b83da1312cebc7d2d1e44a8ff24"} Oct 30 18:21:07.323857 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:07.323454 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-5fc46c9879-vzjtw" event={"ID":"cbcc3d28-dd08-4146-91d5-9138a5057230","Type":"ContainerStarted","Data":"58dd9c55b7ac1c219c0d0e5ec45a6a2cdc6ecc31cd9be04b417893dda1441571"} Oct 30 18:21:07.323857 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:07.323491 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-5fc46c9879-vzjtw" Oct 30 18:21:07.369900 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:07.369828 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-5fc46c9879-vzjtw" podStartSLOduration=2.3698088889999998 podStartE2EDuration="2.369808889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:21:07.367433407 +0000 UTC m=+1719.855070444" watchObservedRunningTime="2026-10-30 18:21:07.369808889 +0000 UTC m=+1719.857445924" Oct 30 18:21:12.500584 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:12.500546 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2"] Oct 30 18:21:12.501070 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:12.500839 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" podUID="82d99e75-d468-4b37-b42e-391602a39c85" containerName="istio-proxy" containerID="cri-o://af862c6c1d1eadc202152dada9a4c2b46f7dd30fde559a5586e11fbcd6592a52" gracePeriod=30 Oct 30 18:21:14.417911 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:14.417827 4455 generic.go:356] "Generic (PLEG): container finished" podID="64c72761-8ae7-47a1-aa47-d10cadeb3e34" containerID="2b69b01b85480b0c4b7f5bf17beb57118c3dde303b369430700f8d0b1ea02ec0" exitCode=1 Oct 30 18:21:14.418277 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:14.417908 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-57f5c66455-l7hkg" event={"ID":"64c72761-8ae7-47a1-aa47-d10cadeb3e34","Type":"ContainerDied","Data":"2b69b01b85480b0c4b7f5bf17beb57118c3dde303b369430700f8d0b1ea02ec0"} Oct 30 18:21:14.418365 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:14.418352 4455 scope.go:122] "RemoveContainer" containerID="2b69b01b85480b0c4b7f5bf17beb57118c3dde303b369430700f8d0b1ea02ec0" Oct 30 18:21:14.434912 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:14.434887 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 18:21:15.427268 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:15.427227 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-57f5c66455-l7hkg" event={"ID":"64c72761-8ae7-47a1-aa47-d10cadeb3e34","Type":"ContainerStarted","Data":"ef2b326f471217999016753c8636f84ff208a59afcb80868e10e6b8d4804d91c"} Oct 30 18:21:15.427797 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:15.427503 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-57f5c66455-l7hkg" Oct 30 18:21:16.183540 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:16.183499 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr"] Oct 30 18:21:16.188357 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:16.188339 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr" Oct 30 18:21:16.217172 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:16.217138 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-e0bc49-serving-cert\"" Oct 30 18:21:16.223069 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:16.223037 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr"] Oct 30 18:21:16.249407 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:16.249372 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a99b4f25-e266-43dd-bd25-576847ed1b4a-maas-api-tls\") pod \"maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr\" (UID: \"a99b4f25-e266-43dd-bd25-576847ed1b4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr" Oct 30 18:21:16.249574 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:16.249529 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a99b4f25-e266-43dd-bd25-576847ed1b4a-openshift-service-ca\") pod \"maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr\" (UID: \"a99b4f25-e266-43dd-bd25-576847ed1b4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr" Oct 30 18:21:16.249714 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:16.249695 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a99b4f25-e266-43dd-bd25-576847ed1b4a-metrics-certs\") pod \"maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr\" (UID: \"a99b4f25-e266-43dd-bd25-576847ed1b4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr" Oct 30 18:21:16.249841 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:16.249828 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-494sp\" (UniqueName: \"kubernetes.io/projected/a99b4f25-e266-43dd-bd25-576847ed1b4a-kube-api-access-494sp\") pod \"maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr\" (UID: \"a99b4f25-e266-43dd-bd25-576847ed1b4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr" Oct 30 18:21:16.351522 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:16.351487 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-494sp\" (UniqueName: \"kubernetes.io/projected/a99b4f25-e266-43dd-bd25-576847ed1b4a-kube-api-access-494sp\") pod \"maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr\" (UID: \"a99b4f25-e266-43dd-bd25-576847ed1b4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr" Oct 30 18:21:16.351727 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:16.351611 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a99b4f25-e266-43dd-bd25-576847ed1b4a-maas-api-tls\") pod \"maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr\" (UID: \"a99b4f25-e266-43dd-bd25-576847ed1b4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr" Oct 30 18:21:16.351727 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:16.351711 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a99b4f25-e266-43dd-bd25-576847ed1b4a-openshift-service-ca\") pod \"maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr\" (UID: \"a99b4f25-e266-43dd-bd25-576847ed1b4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr" Oct 30 18:21:16.351887 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:16.351870 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a99b4f25-e266-43dd-bd25-576847ed1b4a-metrics-certs\") pod \"maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr\" (UID: \"a99b4f25-e266-43dd-bd25-576847ed1b4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr" Oct 30 18:21:16.352906 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:16.352874 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a99b4f25-e266-43dd-bd25-576847ed1b4a-openshift-service-ca\") pod \"maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr\" (UID: \"a99b4f25-e266-43dd-bd25-576847ed1b4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr" Oct 30 18:21:16.354465 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:16.354438 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a99b4f25-e266-43dd-bd25-576847ed1b4a-maas-api-tls\") pod \"maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr\" (UID: \"a99b4f25-e266-43dd-bd25-576847ed1b4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr" Oct 30 18:21:16.354572 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:16.354480 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a99b4f25-e266-43dd-bd25-576847ed1b4a-metrics-certs\") pod \"maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr\" (UID: \"a99b4f25-e266-43dd-bd25-576847ed1b4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr" Oct 30 18:21:16.376905 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:16.376876 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-494sp\" (UniqueName: \"kubernetes.io/projected/a99b4f25-e266-43dd-bd25-576847ed1b4a-kube-api-access-494sp\") pod \"maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr\" (UID: \"a99b4f25-e266-43dd-bd25-576847ed1b4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr" Oct 30 18:21:16.510694 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:16.510656 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr" Oct 30 18:21:16.844169 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:21:16.844139 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda99b4f25_e266_43dd_bd25_576847ed1b4a.slice/crio-2d4f02cc4ca0ea09626888bfe50374ac152291c9c7156a3768a30bf5ea48a20e WatchSource:0}: Error finding container 2d4f02cc4ca0ea09626888bfe50374ac152291c9c7156a3768a30bf5ea48a20e: Status 404 returned error can't find the container with id 2d4f02cc4ca0ea09626888bfe50374ac152291c9c7156a3768a30bf5ea48a20e Oct 30 18:21:16.846569 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:16.846544 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr"] Oct 30 18:21:17.507458 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:17.503859 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr" event={"ID":"a99b4f25-e266-43dd-bd25-576847ed1b4a","Type":"ContainerStarted","Data":"415686924fb73625eeb0794e47272a3f9124bc0d7a53acca13da3c081828c726"} Oct 30 18:21:17.507458 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:17.503931 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr" event={"ID":"a99b4f25-e266-43dd-bd25-576847ed1b4a","Type":"ContainerStarted","Data":"2d4f02cc4ca0ea09626888bfe50374ac152291c9c7156a3768a30bf5ea48a20e"} Oct 30 18:21:17.507458 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:17.505352 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr" Oct 30 18:21:17.553679 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:17.553593 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr" podStartSLOduration=1.2163325409999999 podStartE2EDuration="1.553579782s" totalImagesPullingTime="337.247241ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:21:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:21:17.550095867 +0000 UTC m=+1730.037732901" watchObservedRunningTime="2026-10-30 18:21:17.553579782 +0000 UTC m=+1730.041216815" Oct 30 18:21:17.901757 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:17.901727 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" Oct 30 18:21:18.040526 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.040490 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82d99e75-d468-4b37-b42e-391602a39c85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82d99e75-d468-4b37-b42e-391602a39c85-istiod-ca-cert\") pod \"82d99e75-d468-4b37-b42e-391602a39c85\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " Oct 30 18:21:18.040726 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.040554 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82d99e75-d468-4b37-b42e-391602a39c85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82d99e75-d468-4b37-b42e-391602a39c85-workload-certs\") pod \"82d99e75-d468-4b37-b42e-391602a39c85\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " Oct 30 18:21:18.040726 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.040662 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82d99e75-d468-4b37-b42e-391602a39c85-istio-token\" (UniqueName: \"kubernetes.io/projected/82d99e75-d468-4b37-b42e-391602a39c85-istio-token\") pod \"82d99e75-d468-4b37-b42e-391602a39c85\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " Oct 30 18:21:18.040726 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.040713 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82d99e75-d468-4b37-b42e-391602a39c85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82d99e75-d468-4b37-b42e-391602a39c85-istio-envoy\") pod \"82d99e75-d468-4b37-b42e-391602a39c85\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " Oct 30 18:21:18.040907 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.040745 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82d99e75-d468-4b37-b42e-391602a39c85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82d99e75-d468-4b37-b42e-391602a39c85-istio-data\") pod \"82d99e75-d468-4b37-b42e-391602a39c85\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " Oct 30 18:21:18.040957 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.040911 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82d99e75-d468-4b37-b42e-391602a39c85-kube-api-access-286qq\" (UniqueName: \"kubernetes.io/projected/82d99e75-d468-4b37-b42e-391602a39c85-kube-api-access-286qq\") pod \"82d99e75-d468-4b37-b42e-391602a39c85\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " Oct 30 18:21:18.040957 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.040916 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d99e75-d468-4b37-b42e-391602a39c85-workload-certs" pod "82d99e75-d468-4b37-b42e-391602a39c85" (UID: "82d99e75-d468-4b37-b42e-391602a39c85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:18.040957 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.040940 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82d99e75-d468-4b37-b42e-391602a39c85-istiod-ca-cert" pod "82d99e75-d468-4b37-b42e-391602a39c85" (UID: "82d99e75-d468-4b37-b42e-391602a39c85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:21:18.041129 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.040945 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82d99e75-d468-4b37-b42e-391602a39c85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d99e75-d468-4b37-b42e-391602a39c85-credential-socket\") pod \"82d99e75-d468-4b37-b42e-391602a39c85\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " Oct 30 18:21:18.041129 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.041083 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d99e75-d468-4b37-b42e-391602a39c85-istio-data" pod "82d99e75-d468-4b37-b42e-391602a39c85" (UID: "82d99e75-d468-4b37-b42e-391602a39c85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:18.041129 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.041093 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82d99e75-d468-4b37-b42e-391602a39c85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82d99e75-d468-4b37-b42e-391602a39c85-istio-podinfo\") pod \"82d99e75-d468-4b37-b42e-391602a39c85\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " Oct 30 18:21:18.041298 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.041125 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d99e75-d468-4b37-b42e-391602a39c85-credential-socket" pod "82d99e75-d468-4b37-b42e-391602a39c85" (UID: "82d99e75-d468-4b37-b42e-391602a39c85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:18.041298 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.041150 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82d99e75-d468-4b37-b42e-391602a39c85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d99e75-d468-4b37-b42e-391602a39c85-workload-socket\") pod \"82d99e75-d468-4b37-b42e-391602a39c85\" (UID: \"82d99e75-d468-4b37-b42e-391602a39c85\") " Oct 30 18:21:18.041459 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.041414 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d99e75-d468-4b37-b42e-391602a39c85-workload-socket" pod "82d99e75-d468-4b37-b42e-391602a39c85" (UID: "82d99e75-d468-4b37-b42e-391602a39c85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:18.042160 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.042133 4455 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82d99e75-d468-4b37-b42e-391602a39c85-workload-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:21:18.042272 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.042164 4455 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82d99e75-d468-4b37-b42e-391602a39c85-istio-data\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:21:18.042272 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.042178 4455 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d99e75-d468-4b37-b42e-391602a39c85-credential-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:21:18.042272 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.042191 4455 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d99e75-d468-4b37-b42e-391602a39c85-workload-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:21:18.042272 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.042204 4455 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82d99e75-d468-4b37-b42e-391602a39c85-istiod-ca-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:21:18.043259 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.043237 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82d99e75-d468-4b37-b42e-391602a39c85-istio-token" pod "82d99e75-d468-4b37-b42e-391602a39c85" (UID: "82d99e75-d468-4b37-b42e-391602a39c85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:18.043387 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.043353 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82d99e75-d468-4b37-b42e-391602a39c85-kube-api-access-286qq" pod "82d99e75-d468-4b37-b42e-391602a39c85" (UID: "82d99e75-d468-4b37-b42e-391602a39c85"). InnerVolumeSpecName "kube-api-access-286qq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:18.043688 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.043669 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82d99e75-d468-4b37-b42e-391602a39c85-istio-podinfo" pod "82d99e75-d468-4b37-b42e-391602a39c85" (UID: "82d99e75-d468-4b37-b42e-391602a39c85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:21:18.043688 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.043671 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d99e75-d468-4b37-b42e-391602a39c85-istio-envoy" pod "82d99e75-d468-4b37-b42e-391602a39c85" (UID: "82d99e75-d468-4b37-b42e-391602a39c85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:18.149802 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.149701 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-286qq\" (UniqueName: \"kubernetes.io/projected/82d99e75-d468-4b37-b42e-391602a39c85-kube-api-access-286qq\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:21:18.149802 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.149743 4455 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82d99e75-d468-4b37-b42e-391602a39c85-istio-podinfo\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:21:18.149802 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.149758 4455 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82d99e75-d468-4b37-b42e-391602a39c85-istio-token\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:21:18.149802 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.149772 4455 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82d99e75-d468-4b37-b42e-391602a39c85-istio-envoy\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:21:18.334507 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.334467 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-5fc46c9879-vzjtw" Oct 30 18:21:18.515207 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.515169 4455 generic.go:356] "Generic (PLEG): container finished" podID="82d99e75-d468-4b37-b42e-391602a39c85" containerID="af862c6c1d1eadc202152dada9a4c2b46f7dd30fde559a5586e11fbcd6592a52" exitCode=0 Oct 30 18:21:18.515469 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.515216 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" event={"ID":"82d99e75-d468-4b37-b42e-391602a39c85","Type":"ContainerDied","Data":"af862c6c1d1eadc202152dada9a4c2b46f7dd30fde559a5586e11fbcd6592a52"} Oct 30 18:21:18.515469 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.515253 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" Oct 30 18:21:18.515469 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.515412 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" event={"ID":"82d99e75-d468-4b37-b42e-391602a39c85","Type":"ContainerDied","Data":"75457c30c16efebbad64a2083831ba1c93cf82853f7935b495c957ce5a31c1ab"} Oct 30 18:21:18.515469 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.515443 4455 scope.go:122] "RemoveContainer" containerID="af862c6c1d1eadc202152dada9a4c2b46f7dd30fde559a5586e11fbcd6592a52" Oct 30 18:21:18.528219 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.528197 4455 scope.go:122] "RemoveContainer" containerID="af862c6c1d1eadc202152dada9a4c2b46f7dd30fde559a5586e11fbcd6592a52" Oct 30 18:21:18.528513 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:21:18.528495 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af862c6c1d1eadc202152dada9a4c2b46f7dd30fde559a5586e11fbcd6592a52\": container with ID starting with af862c6c1d1eadc202152dada9a4c2b46f7dd30fde559a5586e11fbcd6592a52 not found: ID does not exist" containerID="af862c6c1d1eadc202152dada9a4c2b46f7dd30fde559a5586e11fbcd6592a52" Oct 30 18:21:18.528607 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.528522 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af862c6c1d1eadc202152dada9a4c2b46f7dd30fde559a5586e11fbcd6592a52"} err="failed to get container status \"af862c6c1d1eadc202152dada9a4c2b46f7dd30fde559a5586e11fbcd6592a52\": rpc error: code = NotFound desc = could not find container \"af862c6c1d1eadc202152dada9a4c2b46f7dd30fde559a5586e11fbcd6592a52\": container with ID starting with af862c6c1d1eadc202152dada9a4c2b46f7dd30fde559a5586e11fbcd6592a52 not found: ID does not exist" Oct 30 18:21:18.578965 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.578928 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2"] Oct 30 18:21:18.585456 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.585429 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2"] Oct 30 18:21:18.772247 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.772143 4455 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9kmd2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:21:18.772247 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:18.772228 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9kmd2" podUID="82d99e75-d468-4b37-b42e-391602a39c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:21:19.636273 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:19.636238 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82d99e75-d468-4b37-b42e-391602a39c85" path="/var/lib/kubelet/pods/82d99e75-d468-4b37-b42e-391602a39c85/volumes" Oct 30 18:21:24.531511 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:24.531482 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr" Oct 30 18:21:24.926215 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:24.926132 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t"] Oct 30 18:21:24.926532 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:24.926480 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t" podUID="347e628a-8625-4f90-8061-8f24241182ee" containerName="maas-api" containerID="cri-o://acd52034a06c0e62e080314e36a3f140eb9c939b7e8ebfa1ba06b8387db75748" gracePeriod=30 Oct 30 18:21:25.087678 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.087647 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-853d3a-5dbcc7fb74-w5g67"] Oct 30 18:21:25.088111 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.088081 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-5dbcc7fb74-w5g67" podUID="8e262b82-061e-4084-a5e9-b133fdfe01da" containerName="payload-processing" containerID="cri-o://f95b640388a89f7f12a8bb90412fa45e57c429e35fde3d506895862cfe68f190" gracePeriod=30 Oct 30 18:21:25.134354 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.133982 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-d5df8bc9-k4tss"] Oct 30 18:21:25.134354 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.134223 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-d5df8bc9-k4tss" podUID="b16e7e76-d4fc-4844-9c0b-767c2ceed45e" containerName="payload-pre-processing" containerID="cri-o://d6b50ea9c64c134569c709726081cef1624d335cf34f5e9d54fdcd890f9a1af2" gracePeriod=30 Oct 30 18:21:25.326404 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.326284 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t" Oct 30 18:21:25.480393 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.425097 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/347e628a-8625-4f90-8061-8f24241182ee-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/347e628a-8625-4f90-8061-8f24241182ee-openshift-service-ca\") pod \"347e628a-8625-4f90-8061-8f24241182ee\" (UID: \"347e628a-8625-4f90-8061-8f24241182ee\") " Oct 30 18:21:25.480393 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.425156 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/347e628a-8625-4f90-8061-8f24241182ee-kube-api-access-plctx\" (UniqueName: \"kubernetes.io/projected/347e628a-8625-4f90-8061-8f24241182ee-kube-api-access-plctx\") pod \"347e628a-8625-4f90-8061-8f24241182ee\" (UID: \"347e628a-8625-4f90-8061-8f24241182ee\") " Oct 30 18:21:25.480393 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.425247 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/347e628a-8625-4f90-8061-8f24241182ee-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/347e628a-8625-4f90-8061-8f24241182ee-maas-api-tls\") pod \"347e628a-8625-4f90-8061-8f24241182ee\" (UID: \"347e628a-8625-4f90-8061-8f24241182ee\") " Oct 30 18:21:25.480393 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.425361 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/347e628a-8625-4f90-8061-8f24241182ee-metrics-certs\" (UniqueName: \"kubernetes.io/secret/347e628a-8625-4f90-8061-8f24241182ee-metrics-certs\") pod \"347e628a-8625-4f90-8061-8f24241182ee\" (UID: \"347e628a-8625-4f90-8061-8f24241182ee\") " Oct 30 18:21:25.480393 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.476794 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/347e628a-8625-4f90-8061-8f24241182ee-openshift-service-ca" pod "347e628a-8625-4f90-8061-8f24241182ee" (UID: "347e628a-8625-4f90-8061-8f24241182ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:21:25.542344 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.528988 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/347e628a-8625-4f90-8061-8f24241182ee-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:21:25.653374 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.653315 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/347e628a-8625-4f90-8061-8f24241182ee-maas-api-tls" pod "347e628a-8625-4f90-8061-8f24241182ee" (UID: "347e628a-8625-4f90-8061-8f24241182ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:21:25.657900 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.657869 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/347e628a-8625-4f90-8061-8f24241182ee-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:21:25.663271 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.659229 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/347e628a-8625-4f90-8061-8f24241182ee-kube-api-access-plctx" pod "347e628a-8625-4f90-8061-8f24241182ee" (UID: "347e628a-8625-4f90-8061-8f24241182ee"). InnerVolumeSpecName "kube-api-access-plctx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:25.692660 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.692000 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/347e628a-8625-4f90-8061-8f24241182ee-metrics-certs" pod "347e628a-8625-4f90-8061-8f24241182ee" (UID: "347e628a-8625-4f90-8061-8f24241182ee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:21:25.713544 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.713512 4455 generic.go:356] "Generic (PLEG): container finished" podID="347e628a-8625-4f90-8061-8f24241182ee" containerID="acd52034a06c0e62e080314e36a3f140eb9c939b7e8ebfa1ba06b8387db75748" exitCode=0 Oct 30 18:21:25.713911 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.713891 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t" Oct 30 18:21:25.730945 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.730871 4455 generic.go:356] "Generic (PLEG): container finished" podID="b16e7e76-d4fc-4844-9c0b-767c2ceed45e" containerID="d6b50ea9c64c134569c709726081cef1624d335cf34f5e9d54fdcd890f9a1af2" exitCode=0 Oct 30 18:21:25.751202 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.751146 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t" event={"ID":"347e628a-8625-4f90-8061-8f24241182ee","Type":"ContainerDied","Data":"acd52034a06c0e62e080314e36a3f140eb9c939b7e8ebfa1ba06b8387db75748"} Oct 30 18:21:25.751384 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.751225 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t" event={"ID":"347e628a-8625-4f90-8061-8f24241182ee","Type":"ContainerDied","Data":"d69f2fea0a8d495c5256d1790e15cb7e63566eb3f3f70f880dd77f637b7e25be"} Oct 30 18:21:25.751384 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.751254 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-d5df8bc9-k4tss" event={"ID":"b16e7e76-d4fc-4844-9c0b-767c2ceed45e","Type":"ContainerDied","Data":"d6b50ea9c64c134569c709726081cef1624d335cf34f5e9d54fdcd890f9a1af2"} Oct 30 18:21:25.751384 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.751251 4455 generic.go:356] "Generic (PLEG): container finished" podID="8e262b82-061e-4084-a5e9-b133fdfe01da" containerID="f95b640388a89f7f12a8bb90412fa45e57c429e35fde3d506895862cfe68f190" exitCode=0 Oct 30 18:21:25.751384 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.751288 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-5dbcc7fb74-w5g67" event={"ID":"8e262b82-061e-4084-a5e9-b133fdfe01da","Type":"ContainerDied","Data":"f95b640388a89f7f12a8bb90412fa45e57c429e35fde3d506895862cfe68f190"} Oct 30 18:21:25.751384 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.751358 4455 scope.go:122] "RemoveContainer" containerID="acd52034a06c0e62e080314e36a3f140eb9c939b7e8ebfa1ba06b8387db75748" Oct 30 18:21:25.752476 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.752278 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-d5df8bc9-k4tss" Oct 30 18:21:25.764383 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.761883 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plctx\" (UniqueName: \"kubernetes.io/projected/347e628a-8625-4f90-8061-8f24241182ee-kube-api-access-plctx\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:21:25.764383 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.761911 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/347e628a-8625-4f90-8061-8f24241182ee-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:21:25.771304 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.771282 4455 scope.go:122] "RemoveContainer" containerID="acd52034a06c0e62e080314e36a3f140eb9c939b7e8ebfa1ba06b8387db75748" Oct 30 18:21:25.771707 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:21:25.771678 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acd52034a06c0e62e080314e36a3f140eb9c939b7e8ebfa1ba06b8387db75748\": container with ID starting with acd52034a06c0e62e080314e36a3f140eb9c939b7e8ebfa1ba06b8387db75748 not found: ID does not exist" containerID="acd52034a06c0e62e080314e36a3f140eb9c939b7e8ebfa1ba06b8387db75748" Oct 30 18:21:25.771787 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.771715 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acd52034a06c0e62e080314e36a3f140eb9c939b7e8ebfa1ba06b8387db75748"} err="failed to get container status \"acd52034a06c0e62e080314e36a3f140eb9c939b7e8ebfa1ba06b8387db75748\": rpc error: code = NotFound desc = could not find container \"acd52034a06c0e62e080314e36a3f140eb9c939b7e8ebfa1ba06b8387db75748\": container with ID starting with acd52034a06c0e62e080314e36a3f140eb9c939b7e8ebfa1ba06b8387db75748 not found: ID does not exist" Oct 30 18:21:25.771787 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.771732 4455 scope.go:122] "RemoveContainer" containerID="5d0d6720550512af9898a55a41880fad8278aa2c1373c5f25164c7b953224ae2" Oct 30 18:21:25.802069 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.802038 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-5dbcc7fb74-w5g67" Oct 30 18:21:25.871346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.867477 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b16e7e76-d4fc-4844-9c0b-767c2ceed45e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b16e7e76-d4fc-4844-9c0b-767c2ceed45e-plugins-config-volume\") pod \"b16e7e76-d4fc-4844-9c0b-767c2ceed45e\" (UID: \"b16e7e76-d4fc-4844-9c0b-767c2ceed45e\") " Oct 30 18:21:25.871346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.867552 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b16e7e76-d4fc-4844-9c0b-767c2ceed45e-kube-api-access-tvvp8\" (UniqueName: \"kubernetes.io/projected/b16e7e76-d4fc-4844-9c0b-767c2ceed45e-kube-api-access-tvvp8\") pod \"b16e7e76-d4fc-4844-9c0b-767c2ceed45e\" (UID: \"b16e7e76-d4fc-4844-9c0b-767c2ceed45e\") " Oct 30 18:21:25.912393 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.891549 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b16e7e76-d4fc-4844-9c0b-767c2ceed45e-plugins-config-volume" pod "b16e7e76-d4fc-4844-9c0b-767c2ceed45e" (UID: "b16e7e76-d4fc-4844-9c0b-767c2ceed45e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:21:25.912393 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.897127 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b16e7e76-d4fc-4844-9c0b-767c2ceed45e-kube-api-access-tvvp8" pod "b16e7e76-d4fc-4844-9c0b-767c2ceed45e" (UID: "b16e7e76-d4fc-4844-9c0b-767c2ceed45e"). InnerVolumeSpecName "kube-api-access-tvvp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:25.937347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.923227 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t"] Oct 30 18:21:25.937347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.932157 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-853d3a-6f85649995-lkv6t"] Oct 30 18:21:25.969430 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.969402 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e262b82-061e-4084-a5e9-b133fdfe01da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e262b82-061e-4084-a5e9-b133fdfe01da-plugins-config-volume\") pod \"8e262b82-061e-4084-a5e9-b133fdfe01da\" (UID: \"8e262b82-061e-4084-a5e9-b133fdfe01da\") " Oct 30 18:21:25.969591 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.969575 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e262b82-061e-4084-a5e9-b133fdfe01da-kube-api-access-c6lht\" (UniqueName: \"kubernetes.io/projected/8e262b82-061e-4084-a5e9-b133fdfe01da-kube-api-access-c6lht\") pod \"8e262b82-061e-4084-a5e9-b133fdfe01da\" (UID: \"8e262b82-061e-4084-a5e9-b133fdfe01da\") " Oct 30 18:21:25.969869 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.969841 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e262b82-061e-4084-a5e9-b133fdfe01da-plugins-config-volume" pod "8e262b82-061e-4084-a5e9-b133fdfe01da" (UID: "8e262b82-061e-4084-a5e9-b133fdfe01da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:21:25.970429 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.970363 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b16e7e76-d4fc-4844-9c0b-767c2ceed45e-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:21:25.970429 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.970390 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvvp8\" (UniqueName: \"kubernetes.io/projected/b16e7e76-d4fc-4844-9c0b-767c2ceed45e-kube-api-access-tvvp8\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:21:25.970429 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.970404 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e262b82-061e-4084-a5e9-b133fdfe01da-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:21:25.972063 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:25.972034 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e262b82-061e-4084-a5e9-b133fdfe01da-kube-api-access-c6lht" pod "8e262b82-061e-4084-a5e9-b133fdfe01da" (UID: "8e262b82-061e-4084-a5e9-b133fdfe01da"). InnerVolumeSpecName "kube-api-access-c6lht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:26.072481 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:26.072443 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6lht\" (UniqueName: \"kubernetes.io/projected/8e262b82-061e-4084-a5e9-b133fdfe01da-kube-api-access-c6lht\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:21:26.460954 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:26.460875 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-57f5c66455-l7hkg" Oct 30 18:21:26.761649 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:26.761623 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-d5df8bc9-k4tss" Oct 30 18:21:26.762137 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:26.761644 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-d5df8bc9-k4tss" event={"ID":"b16e7e76-d4fc-4844-9c0b-767c2ceed45e","Type":"ContainerDied","Data":"25b8beccaaaaf85dd125307938d8c528958a2b2efae15f159aff7aa47529d299"} Oct 30 18:21:26.762137 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:26.761714 4455 scope.go:122] "RemoveContainer" containerID="d6b50ea9c64c134569c709726081cef1624d335cf34f5e9d54fdcd890f9a1af2" Oct 30 18:21:26.763415 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:26.763393 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-5dbcc7fb74-w5g67" Oct 30 18:21:26.763558 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:26.763410 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-853d3a-5dbcc7fb74-w5g67" event={"ID":"8e262b82-061e-4084-a5e9-b133fdfe01da","Type":"ContainerDied","Data":"d68ea5147d17a0e3a19b6f817bf07d52d3c73ae215694f476fa2edfa7577385c"} Oct 30 18:21:26.775982 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:26.775959 4455 scope.go:122] "RemoveContainer" containerID="f95b640388a89f7f12a8bb90412fa45e57c429e35fde3d506895862cfe68f190" Oct 30 18:21:26.873068 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:26.873033 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-853d3a-5dbcc7fb74-w5g67"] Oct 30 18:21:26.888042 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:26.888011 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-853d3a-5dbcc7fb74-w5g67"] Oct 30 18:21:26.925086 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:26.925045 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-d5df8bc9-k4tss"] Oct 30 18:21:26.929057 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:26.929028 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-853d3a-d5df8bc9-k4tss"] Oct 30 18:21:27.637981 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:27.637944 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="347e628a-8625-4f90-8061-8f24241182ee" path="/var/lib/kubelet/pods/347e628a-8625-4f90-8061-8f24241182ee/volumes" Oct 30 18:21:27.638580 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:27.638558 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e262b82-061e-4084-a5e9-b133fdfe01da" path="/var/lib/kubelet/pods/8e262b82-061e-4084-a5e9-b133fdfe01da/volumes" Oct 30 18:21:27.639022 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:27.639003 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b16e7e76-d4fc-4844-9c0b-767c2ceed45e" path="/var/lib/kubelet/pods/b16e7e76-d4fc-4844-9c0b-767c2ceed45e/volumes" Oct 30 18:21:30.386969 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:21:30.386929 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:21:35.760485 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:35.760411 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-758c6cf78c-glwf2"] Oct 30 18:21:35.762137 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:35.762114 4455 cpu_manager.go:414] "containerMap: removing container" podUID="82d99e75-d468-4b37-b42e-391602a39c85" containerName="istio-proxy" Oct 30 18:21:35.762282 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:35.762272 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82d99e75-d468-4b37-b42e-391602a39c85" containerName="istio-proxy" Oct 30 18:21:35.762365 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:35.762357 4455 cpu_manager.go:414] "containerMap: removing container" podUID="347e628a-8625-4f90-8061-8f24241182ee" containerName="maas-api" Oct 30 18:21:35.762427 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:35.762418 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="347e628a-8625-4f90-8061-8f24241182ee" containerName="maas-api" Oct 30 18:21:35.762501 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:35.762494 4455 cpu_manager.go:414] "containerMap: removing container" podUID="8e262b82-061e-4084-a5e9-b133fdfe01da" containerName="payload-processing" Oct 30 18:21:35.762547 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:35.762540 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e262b82-061e-4084-a5e9-b133fdfe01da" containerName="payload-processing" Oct 30 18:21:35.762618 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:35.762612 4455 cpu_manager.go:414] "containerMap: removing container" podUID="b16e7e76-d4fc-4844-9c0b-767c2ceed45e" containerName="payload-pre-processing" Oct 30 18:21:35.762664 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:35.762657 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b16e7e76-d4fc-4844-9c0b-767c2ceed45e" containerName="payload-pre-processing" Oct 30 18:21:35.762849 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:35.762842 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="82d99e75-d468-4b37-b42e-391602a39c85" containerName="istio-proxy" Oct 30 18:21:35.762919 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:35.762913 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="347e628a-8625-4f90-8061-8f24241182ee" containerName="maas-api" Oct 30 18:21:35.762971 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:35.762965 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e262b82-061e-4084-a5e9-b133fdfe01da" containerName="payload-processing" Oct 30 18:21:35.763017 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:35.763012 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="b16e7e76-d4fc-4844-9c0b-767c2ceed45e" containerName="payload-pre-processing" Oct 30 18:21:35.763067 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:35.763061 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e262b82-061e-4084-a5e9-b133fdfe01da" containerName="payload-processing" Oct 30 18:21:35.769827 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:35.769806 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-758c6cf78c-glwf2"] Oct 30 18:21:35.769987 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:35.769911 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-758c6cf78c-glwf2" Oct 30 18:21:35.790879 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:35.790835 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-394a31-dockercfg-k4bmz\"" Oct 30 18:21:35.791257 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:35.791213 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-394a31\"" Oct 30 18:21:35.796186 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:35.793011 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-394a31-7d9db95ddb-8xl6n"] Oct 30 18:21:35.796186 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:35.794152 4455 cpu_manager.go:414] "containerMap: removing container" podUID="8e262b82-061e-4084-a5e9-b133fdfe01da" containerName="payload-processing" Oct 30 18:21:35.796186 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:35.794171 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e262b82-061e-4084-a5e9-b133fdfe01da" containerName="payload-processing" Oct 30 18:21:35.857349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:35.824644 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-394a31-7d9db95ddb-8xl6n"] Oct 30 18:21:35.857349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:35.824761 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-7d9db95ddb-8xl6n" Oct 30 18:21:35.907873 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:35.900594 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwpch\" (UniqueName: \"kubernetes.io/projected/d9e693c8-eff4-4afd-8c2a-3009b6c0c27f-kube-api-access-zwpch\") pod \"payload-pre-processing-e2e-ipp-b-394a31-758c6cf78c-glwf2\" (UID: \"d9e693c8-eff4-4afd-8c2a-3009b6c0c27f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-758c6cf78c-glwf2" Oct 30 18:21:35.907873 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:35.900726 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj9nb\" (UniqueName: \"kubernetes.io/projected/97e9aed5-3f20-4301-b832-a78b35c9fb71-kube-api-access-hj9nb\") pod \"payload-processing-e2e-ipp-b-394a31-7d9db95ddb-8xl6n\" (UID: \"97e9aed5-3f20-4301-b832-a78b35c9fb71\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-7d9db95ddb-8xl6n" Oct 30 18:21:35.907873 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:35.900801 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97e9aed5-3f20-4301-b832-a78b35c9fb71-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-394a31-7d9db95ddb-8xl6n\" (UID: \"97e9aed5-3f20-4301-b832-a78b35c9fb71\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-7d9db95ddb-8xl6n" Oct 30 18:21:35.907873 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:35.900883 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e693c8-eff4-4afd-8c2a-3009b6c0c27f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-394a31-758c6cf78c-glwf2\" (UID: \"d9e693c8-eff4-4afd-8c2a-3009b6c0c27f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-758c6cf78c-glwf2" Oct 30 18:21:36.002146 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:36.002091 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwpch\" (UniqueName: \"kubernetes.io/projected/d9e693c8-eff4-4afd-8c2a-3009b6c0c27f-kube-api-access-zwpch\") pod \"payload-pre-processing-e2e-ipp-b-394a31-758c6cf78c-glwf2\" (UID: \"d9e693c8-eff4-4afd-8c2a-3009b6c0c27f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-758c6cf78c-glwf2" Oct 30 18:21:36.002503 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:36.002474 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj9nb\" (UniqueName: \"kubernetes.io/projected/97e9aed5-3f20-4301-b832-a78b35c9fb71-kube-api-access-hj9nb\") pod \"payload-processing-e2e-ipp-b-394a31-7d9db95ddb-8xl6n\" (UID: \"97e9aed5-3f20-4301-b832-a78b35c9fb71\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-7d9db95ddb-8xl6n" Oct 30 18:21:36.002646 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:36.002547 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97e9aed5-3f20-4301-b832-a78b35c9fb71-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-394a31-7d9db95ddb-8xl6n\" (UID: \"97e9aed5-3f20-4301-b832-a78b35c9fb71\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-7d9db95ddb-8xl6n" Oct 30 18:21:36.002646 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:36.002623 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e693c8-eff4-4afd-8c2a-3009b6c0c27f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-394a31-758c6cf78c-glwf2\" (UID: \"d9e693c8-eff4-4afd-8c2a-3009b6c0c27f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-758c6cf78c-glwf2" Oct 30 18:21:36.003467 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:36.003445 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97e9aed5-3f20-4301-b832-a78b35c9fb71-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-394a31-7d9db95ddb-8xl6n\" (UID: \"97e9aed5-3f20-4301-b832-a78b35c9fb71\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-7d9db95ddb-8xl6n" Oct 30 18:21:36.003593 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:36.003572 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e693c8-eff4-4afd-8c2a-3009b6c0c27f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-394a31-758c6cf78c-glwf2\" (UID: \"d9e693c8-eff4-4afd-8c2a-3009b6c0c27f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-758c6cf78c-glwf2" Oct 30 18:21:36.027196 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:36.027140 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj9nb\" (UniqueName: \"kubernetes.io/projected/97e9aed5-3f20-4301-b832-a78b35c9fb71-kube-api-access-hj9nb\") pod \"payload-processing-e2e-ipp-b-394a31-7d9db95ddb-8xl6n\" (UID: \"97e9aed5-3f20-4301-b832-a78b35c9fb71\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-7d9db95ddb-8xl6n" Oct 30 18:21:36.027458 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:36.027434 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwpch\" (UniqueName: \"kubernetes.io/projected/d9e693c8-eff4-4afd-8c2a-3009b6c0c27f-kube-api-access-zwpch\") pod \"payload-pre-processing-e2e-ipp-b-394a31-758c6cf78c-glwf2\" (UID: \"d9e693c8-eff4-4afd-8c2a-3009b6c0c27f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-758c6cf78c-glwf2" Oct 30 18:21:36.110428 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:36.110398 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-758c6cf78c-glwf2" Oct 30 18:21:36.237070 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:36.237030 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-7d9db95ddb-8xl6n" Oct 30 18:21:36.474341 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:36.450745 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-758c6cf78c-glwf2"] Oct 30 18:21:36.589020 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:36.588852 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-394a31-7d9db95ddb-8xl6n"] Oct 30 18:21:36.988506 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:36.988467 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-758c6cf78c-glwf2" event={"ID":"d9e693c8-eff4-4afd-8c2a-3009b6c0c27f","Type":"ContainerStarted","Data":"140b5d1fcb6bbbb0e9677d7f92db6f17a0dd2df891f5ff981904cb3ae1e352c2"} Oct 30 18:21:36.988967 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:36.988535 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-758c6cf78c-glwf2" event={"ID":"d9e693c8-eff4-4afd-8c2a-3009b6c0c27f","Type":"ContainerStarted","Data":"c6f0bc0b701fcad0f7d42a6cf22ec32fdd1c02afe66d7fba7299aadc7925151d"} Oct 30 18:21:36.988967 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:36.988567 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-758c6cf78c-glwf2" Oct 30 18:21:36.990492 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:36.990466 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-7d9db95ddb-8xl6n" event={"ID":"97e9aed5-3f20-4301-b832-a78b35c9fb71","Type":"ContainerStarted","Data":"ad11f172ba6b49e3b39780038ab7e5c2b1f1401488cd72dc88b57b4ae4f01464"} Oct 30 18:21:36.990638 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:36.990515 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-7d9db95ddb-8xl6n" Oct 30 18:21:36.990638 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:36.990532 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-7d9db95ddb-8xl6n" event={"ID":"97e9aed5-3f20-4301-b832-a78b35c9fb71","Type":"ContainerStarted","Data":"a49f6d06ee16ca3532b9ca6bd25af44bc7a74ea7bda2928e838ee406fb4c55a2"} Oct 30 18:21:37.038706 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:37.038609 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-758c6cf78c-glwf2" podStartSLOduration=2.038592891 podStartE2EDuration="2.038592891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:21:37.032114399 +0000 UTC m=+1749.519751436" watchObservedRunningTime="2026-10-30 18:21:37.038592891 +0000 UTC m=+1749.526229927" Oct 30 18:21:37.077518 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:37.077457 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-7d9db95ddb-8xl6n" podStartSLOduration=2.077439795 podStartE2EDuration="2.077439795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:21:37.077028697 +0000 UTC m=+1749.564665732" watchObservedRunningTime="2026-10-30 18:21:37.077439795 +0000 UTC m=+1749.565076830" Oct 30 18:21:46.888498 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:46.888459 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v"] Oct 30 18:21:46.888998 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:46.888714 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v" podUID="22a3bbc2-f6ee-409d-937b-a0faf2903240" containerName="maas-api" containerID="cri-o://e11b301b3129ae62b37873f32f59d745e2faed3caac9a1cc190f424d54db2ca8" gracePeriod=30 Oct 30 18:21:47.090046 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:47.090005 4455 generic.go:356] "Generic (PLEG): container finished" podID="22a3bbc2-f6ee-409d-937b-a0faf2903240" containerID="e11b301b3129ae62b37873f32f59d745e2faed3caac9a1cc190f424d54db2ca8" exitCode=0 Oct 30 18:21:47.090264 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:47.090245 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v" event={"ID":"22a3bbc2-f6ee-409d-937b-a0faf2903240","Type":"ContainerDied","Data":"e11b301b3129ae62b37873f32f59d745e2faed3caac9a1cc190f424d54db2ca8"} Oct 30 18:21:47.093210 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:47.093181 4455 generic.go:356] "Generic (PLEG): container finished" podID="97e9aed5-3f20-4301-b832-a78b35c9fb71" containerID="ad11f172ba6b49e3b39780038ab7e5c2b1f1401488cd72dc88b57b4ae4f01464" exitCode=1 Oct 30 18:21:47.093412 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:47.093307 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-7d9db95ddb-8xl6n" event={"ID":"97e9aed5-3f20-4301-b832-a78b35c9fb71","Type":"ContainerDied","Data":"ad11f172ba6b49e3b39780038ab7e5c2b1f1401488cd72dc88b57b4ae4f01464"} Oct 30 18:21:47.098915 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:47.097583 4455 scope.go:122] "RemoveContainer" containerID="ad11f172ba6b49e3b39780038ab7e5c2b1f1401488cd72dc88b57b4ae4f01464" Oct 30 18:21:47.261805 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:47.261736 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v" Oct 30 18:21:47.363308 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:47.363265 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22a3bbc2-f6ee-409d-937b-a0faf2903240-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22a3bbc2-f6ee-409d-937b-a0faf2903240-maas-api-tls\") pod \"22a3bbc2-f6ee-409d-937b-a0faf2903240\" (UID: \"22a3bbc2-f6ee-409d-937b-a0faf2903240\") " Oct 30 18:21:47.363631 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:47.363510 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22a3bbc2-f6ee-409d-937b-a0faf2903240-metrics-certs\" (UniqueName: \"kubernetes.io/secret/22a3bbc2-f6ee-409d-937b-a0faf2903240-metrics-certs\") pod \"22a3bbc2-f6ee-409d-937b-a0faf2903240\" (UID: \"22a3bbc2-f6ee-409d-937b-a0faf2903240\") " Oct 30 18:21:47.363955 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:47.363923 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22a3bbc2-f6ee-409d-937b-a0faf2903240-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a3bbc2-f6ee-409d-937b-a0faf2903240-openshift-service-ca\") pod \"22a3bbc2-f6ee-409d-937b-a0faf2903240\" (UID: \"22a3bbc2-f6ee-409d-937b-a0faf2903240\") " Oct 30 18:21:47.364088 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:47.364013 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22a3bbc2-f6ee-409d-937b-a0faf2903240-kube-api-access-j7fgr\" (UniqueName: \"kubernetes.io/projected/22a3bbc2-f6ee-409d-937b-a0faf2903240-kube-api-access-j7fgr\") pod \"22a3bbc2-f6ee-409d-937b-a0faf2903240\" (UID: \"22a3bbc2-f6ee-409d-937b-a0faf2903240\") " Oct 30 18:21:47.364869 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:47.364837 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22a3bbc2-f6ee-409d-937b-a0faf2903240-openshift-service-ca" pod "22a3bbc2-f6ee-409d-937b-a0faf2903240" (UID: "22a3bbc2-f6ee-409d-937b-a0faf2903240"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:21:47.366557 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:47.366526 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22a3bbc2-f6ee-409d-937b-a0faf2903240-maas-api-tls" pod "22a3bbc2-f6ee-409d-937b-a0faf2903240" (UID: "22a3bbc2-f6ee-409d-937b-a0faf2903240"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:21:47.367261 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:47.367232 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22a3bbc2-f6ee-409d-937b-a0faf2903240-kube-api-access-j7fgr" pod "22a3bbc2-f6ee-409d-937b-a0faf2903240" (UID: "22a3bbc2-f6ee-409d-937b-a0faf2903240"). InnerVolumeSpecName "kube-api-access-j7fgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:47.367261 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:47.367242 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22a3bbc2-f6ee-409d-937b-a0faf2903240-metrics-certs" pod "22a3bbc2-f6ee-409d-937b-a0faf2903240" (UID: "22a3bbc2-f6ee-409d-937b-a0faf2903240"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:21:47.518268 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:47.493692 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22a3bbc2-f6ee-409d-937b-a0faf2903240-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:21:47.518268 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:47.493736 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22a3bbc2-f6ee-409d-937b-a0faf2903240-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:21:47.518268 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:47.493755 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a3bbc2-f6ee-409d-937b-a0faf2903240-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:21:47.518268 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:47.493773 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7fgr\" (UniqueName: \"kubernetes.io/projected/22a3bbc2-f6ee-409d-937b-a0faf2903240-kube-api-access-j7fgr\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:21:48.000505 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:48.000465 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-758c6cf78c-glwf2" podUID="d9e693c8-eff4-4afd-8c2a-3009b6c0c27f" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.150:9004: connect: connection refused" Oct 30 18:21:48.103860 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:48.103824 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-7d9db95ddb-8xl6n" event={"ID":"97e9aed5-3f20-4301-b832-a78b35c9fb71","Type":"ContainerStarted","Data":"daa58f58910292e97b52bb62ac06f9c31d2e5d54231a4be97b68270fb0e8b9ef"} Oct 30 18:21:48.104268 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:48.104238 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-7d9db95ddb-8xl6n" Oct 30 18:21:48.105703 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:48.105680 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v" Oct 30 18:21:48.105844 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:48.105708 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v" event={"ID":"22a3bbc2-f6ee-409d-937b-a0faf2903240","Type":"ContainerDied","Data":"66773b0edc2884afca923a9ccb2fcb1eb2b565bb3bddef0c64b9491dd82fa539"} Oct 30 18:21:48.105844 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:48.105765 4455 scope.go:122] "RemoveContainer" containerID="e11b301b3129ae62b37873f32f59d745e2faed3caac9a1cc190f424d54db2ca8" Oct 30 18:21:48.190838 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:48.190806 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v"] Oct 30 18:21:48.198944 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:48.198913 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c33ee4-77d76696bd-z6s2v"] Oct 30 18:21:49.635879 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:49.635847 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22a3bbc2-f6ee-409d-937b-a0faf2903240" path="/var/lib/kubelet/pods/22a3bbc2-f6ee-409d-937b-a0faf2903240/volumes" Oct 30 18:21:58.000407 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:58.000372 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-758c6cf78c-glwf2" Oct 30 18:21:59.130881 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:21:59.130851 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-7d9db95ddb-8xl6n" Oct 30 18:22:00.630036 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:00.628209 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:22:00.738250 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:22:00.735556 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:22:01.243591 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:01.242428 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:22:01.243591 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:01.242510 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" podUID="1d3471dc-aaf3-4060-a64d-2413e522e65a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:01.661426 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:01.636422 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:01.661426 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:01.636969 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:22:02.826383 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:02.804791 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:03.228808 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:03.228768 4455 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:22:03.229113 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:03.229082 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" podUID="d4e8e077-c9f7-44c8-9dd5-d1c7a2403862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:03.589630 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:03.589535 4455 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:22:03.607820 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:03.607770 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" podUID="e972725e-489d-4f2a-a734-27d7eb570858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:03.815713 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:03.815643 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:04.269542 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:04.269490 4455 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:22:04.291917 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:04.269825 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" podUID="ac80ccea-a611-4de1-951c-479ad3d827a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:04.357359 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:04.357292 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9" podUID="484dd1d5-ee6a-4423-87c4-efbb4ba01767" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:04.861366 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:04.850369 4455 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:22:04.861366 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:04.850482 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" podUID="93b710b6-b5a8-4fe0-ac70-861dd036b70d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:05.200642 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:05.199365 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:06.207247 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:06.205431 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:22:06.265665 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:06.265631 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:22:07.064355 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:07.059076 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q"] Oct 30 18:22:07.131056 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:07.130997 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q" podUID="f4cf887a-fbe0-4738-b919-5110b2f77a7b" containerName="maas-api" containerID="cri-o://e41ef75fae079c585b0ccffdf63aaf0ebb7517e390219fd5d5eb359124415c74" gracePeriod=30 Oct 30 18:22:07.789813 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:22:07.789764 4455 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4cf887a_fbe0_4738_b919_5110b2f77a7b.slice/crio-conmon-e41ef75fae079c585b0ccffdf63aaf0ebb7517e390219fd5d5eb359124415c74.scope\": RecentStats: unable to find data in memory cache]" Oct 30 18:22:07.803211 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:22:07.803159 4455 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4cf887a_fbe0_4738_b919_5110b2f77a7b.slice/crio-conmon-e41ef75fae079c585b0ccffdf63aaf0ebb7517e390219fd5d5eb359124415c74.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4cf887a_fbe0_4738_b919_5110b2f77a7b.slice/crio-e41ef75fae079c585b0ccffdf63aaf0ebb7517e390219fd5d5eb359124415c74.scope\": RecentStats: unable to find data in memory cache]" Oct 30 18:22:08.001349 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:22:07.977612 4455 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4cf887a_fbe0_4738_b919_5110b2f77a7b.slice/crio-conmon-e41ef75fae079c585b0ccffdf63aaf0ebb7517e390219fd5d5eb359124415c74.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4cf887a_fbe0_4738_b919_5110b2f77a7b.slice/crio-e41ef75fae079c585b0ccffdf63aaf0ebb7517e390219fd5d5eb359124415c74.scope\": RecentStats: unable to find data in memory cache]" Oct 30 18:22:08.403735 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:08.403645 4455 generic.go:356] "Generic (PLEG): container finished" podID="f4cf887a-fbe0-4738-b919-5110b2f77a7b" containerID="e41ef75fae079c585b0ccffdf63aaf0ebb7517e390219fd5d5eb359124415c74" exitCode=0 Oct 30 18:22:08.403972 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:08.403955 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q" event={"ID":"f4cf887a-fbe0-4738-b919-5110b2f77a7b","Type":"ContainerDied","Data":"e41ef75fae079c585b0ccffdf63aaf0ebb7517e390219fd5d5eb359124415c74"} Oct 30 18:22:08.494826 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:08.494789 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q" Oct 30 18:22:08.603610 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:08.603582 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4cf887a-fbe0-4738-b919-5110b2f77a7b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4cf887a-fbe0-4738-b919-5110b2f77a7b-maas-api-tls\") pod \"f4cf887a-fbe0-4738-b919-5110b2f77a7b\" (UID: \"f4cf887a-fbe0-4738-b919-5110b2f77a7b\") " Oct 30 18:22:08.603810 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:08.603640 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4cf887a-fbe0-4738-b919-5110b2f77a7b-kube-api-access-fwmvz\" (UniqueName: \"kubernetes.io/projected/f4cf887a-fbe0-4738-b919-5110b2f77a7b-kube-api-access-fwmvz\") pod \"f4cf887a-fbe0-4738-b919-5110b2f77a7b\" (UID: \"f4cf887a-fbe0-4738-b919-5110b2f77a7b\") " Oct 30 18:22:08.603810 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:08.603677 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4cf887a-fbe0-4738-b919-5110b2f77a7b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4cf887a-fbe0-4738-b919-5110b2f77a7b-metrics-certs\") pod \"f4cf887a-fbe0-4738-b919-5110b2f77a7b\" (UID: \"f4cf887a-fbe0-4738-b919-5110b2f77a7b\") " Oct 30 18:22:08.603947 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:08.603837 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4cf887a-fbe0-4738-b919-5110b2f77a7b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4cf887a-fbe0-4738-b919-5110b2f77a7b-openshift-service-ca\") pod \"f4cf887a-fbe0-4738-b919-5110b2f77a7b\" (UID: \"f4cf887a-fbe0-4738-b919-5110b2f77a7b\") " Oct 30 18:22:08.604207 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:08.604172 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4cf887a-fbe0-4738-b919-5110b2f77a7b-openshift-service-ca" pod "f4cf887a-fbe0-4738-b919-5110b2f77a7b" (UID: "f4cf887a-fbe0-4738-b919-5110b2f77a7b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:08.605285 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:08.605253 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4cf887a-fbe0-4738-b919-5110b2f77a7b-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:22:08.606084 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:08.606058 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4cf887a-fbe0-4738-b919-5110b2f77a7b-kube-api-access-fwmvz" pod "f4cf887a-fbe0-4738-b919-5110b2f77a7b" (UID: "f4cf887a-fbe0-4738-b919-5110b2f77a7b"). InnerVolumeSpecName "kube-api-access-fwmvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:08.606295 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:08.606267 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4cf887a-fbe0-4738-b919-5110b2f77a7b-maas-api-tls" pod "f4cf887a-fbe0-4738-b919-5110b2f77a7b" (UID: "f4cf887a-fbe0-4738-b919-5110b2f77a7b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:22:08.606515 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:08.606494 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4cf887a-fbe0-4738-b919-5110b2f77a7b-metrics-certs" pod "f4cf887a-fbe0-4738-b919-5110b2f77a7b" (UID: "f4cf887a-fbe0-4738-b919-5110b2f77a7b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:22:08.706292 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:08.706212 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4cf887a-fbe0-4738-b919-5110b2f77a7b-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:22:08.706292 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:08.706241 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwmvz\" (UniqueName: \"kubernetes.io/projected/f4cf887a-fbe0-4738-b919-5110b2f77a7b-kube-api-access-fwmvz\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:22:08.706292 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:08.706252 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4cf887a-fbe0-4738-b919-5110b2f77a7b-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:22:09.423349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:09.422987 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q" event={"ID":"f4cf887a-fbe0-4738-b919-5110b2f77a7b","Type":"ContainerDied","Data":"1b2452f8d8a3a9e4f0c4f0ad4cef0cea7724f997272b1d4c6c168f5f91fb8da1"} Oct 30 18:22:09.423349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:09.423062 4455 scope.go:122] "RemoveContainer" containerID="e41ef75fae079c585b0ccffdf63aaf0ebb7517e390219fd5d5eb359124415c74" Oct 30 18:22:09.423349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:09.423241 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q" Oct 30 18:22:09.509349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:09.498981 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q"] Oct 30 18:22:09.529761 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:09.525993 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q"] Oct 30 18:22:09.636900 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:09.636867 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4cf887a-fbe0-4738-b919-5110b2f77a7b" path="/var/lib/kubelet/pods/f4cf887a-fbe0-4738-b919-5110b2f77a7b/volumes" Oct 30 18:22:11.334131 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:11.334083 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-189cda-cdcb6df74-wb66q" podUID="f4cf887a-fbe0-4738-b919-5110b2f77a7b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.142:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:12.104635 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:12.104606 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-9l4x8_909b1ff6-6d4a-4887-9b59-efe2806ac0e8/payload-processing/0.log" Oct 30 18:22:16.160375 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:16.160303 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk"] Oct 30 18:22:16.160820 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:16.160599 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" podUID="2bc3ad56-5026-4312-85fb-0f40baceeee9" containerName="istio-proxy" containerID="cri-o://7673c400619052abf4ce86bec66307cfc7501390202d5aca35f03e2900ce264c" gracePeriod=30 Oct 30 18:22:17.605914 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:17.605878 4455 patch_prober.go:29] interesting pod/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:22:17.606301 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:17.605944 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" podUID="2bc3ad56-5026-4312-85fb-0f40baceeee9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:22:19.981655 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:19.981620 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a3be60-cd57b57d7-fnl2f"] Oct 30 18:22:21.475750 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.475720 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" Oct 30 18:22:21.495792 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.495755 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bc3ad56-5026-4312-85fb-0f40baceeee9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bc3ad56-5026-4312-85fb-0f40baceeee9-istio-data\") pod \"2bc3ad56-5026-4312-85fb-0f40baceeee9\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " Oct 30 18:22:21.495986 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.495836 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bc3ad56-5026-4312-85fb-0f40baceeee9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc3ad56-5026-4312-85fb-0f40baceeee9-workload-socket\") pod \"2bc3ad56-5026-4312-85fb-0f40baceeee9\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " Oct 30 18:22:21.495986 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.495942 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bc3ad56-5026-4312-85fb-0f40baceeee9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc3ad56-5026-4312-85fb-0f40baceeee9-credential-socket\") pod \"2bc3ad56-5026-4312-85fb-0f40baceeee9\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " Oct 30 18:22:21.496109 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.496068 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bc3ad56-5026-4312-85fb-0f40baceeee9-istio-token\" (UniqueName: \"kubernetes.io/projected/2bc3ad56-5026-4312-85fb-0f40baceeee9-istio-token\") pod \"2bc3ad56-5026-4312-85fb-0f40baceeee9\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " Oct 30 18:22:21.496169 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.496109 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2bc3ad56-5026-4312-85fb-0f40baceeee9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bc3ad56-5026-4312-85fb-0f40baceeee9-istio-podinfo\") pod \"2bc3ad56-5026-4312-85fb-0f40baceeee9\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " Oct 30 18:22:21.496169 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.496139 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bc3ad56-5026-4312-85fb-0f40baceeee9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bc3ad56-5026-4312-85fb-0f40baceeee9-workload-certs\") pod \"2bc3ad56-5026-4312-85fb-0f40baceeee9\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " Oct 30 18:22:21.496296 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.496189 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bc3ad56-5026-4312-85fb-0f40baceeee9-kube-api-access-2v9hz\" (UniqueName: \"kubernetes.io/projected/2bc3ad56-5026-4312-85fb-0f40baceeee9-kube-api-access-2v9hz\") pod \"2bc3ad56-5026-4312-85fb-0f40baceeee9\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " Oct 30 18:22:21.496296 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.496219 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bc3ad56-5026-4312-85fb-0f40baceeee9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bc3ad56-5026-4312-85fb-0f40baceeee9-istio-envoy\") pod \"2bc3ad56-5026-4312-85fb-0f40baceeee9\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " Oct 30 18:22:21.496296 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.496231 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bc3ad56-5026-4312-85fb-0f40baceeee9-workload-socket" pod "2bc3ad56-5026-4312-85fb-0f40baceeee9" (UID: "2bc3ad56-5026-4312-85fb-0f40baceeee9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:21.496486 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.496405 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bc3ad56-5026-4312-85fb-0f40baceeee9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bc3ad56-5026-4312-85fb-0f40baceeee9-istiod-ca-cert\") pod \"2bc3ad56-5026-4312-85fb-0f40baceeee9\" (UID: \"2bc3ad56-5026-4312-85fb-0f40baceeee9\") " Oct 30 18:22:21.496549 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.496521 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bc3ad56-5026-4312-85fb-0f40baceeee9-workload-certs" pod "2bc3ad56-5026-4312-85fb-0f40baceeee9" (UID: "2bc3ad56-5026-4312-85fb-0f40baceeee9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:21.496921 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.496652 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bc3ad56-5026-4312-85fb-0f40baceeee9-istio-data" pod "2bc3ad56-5026-4312-85fb-0f40baceeee9" (UID: "2bc3ad56-5026-4312-85fb-0f40baceeee9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:21.496921 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.496812 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bc3ad56-5026-4312-85fb-0f40baceeee9-credential-socket" pod "2bc3ad56-5026-4312-85fb-0f40baceeee9" (UID: "2bc3ad56-5026-4312-85fb-0f40baceeee9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:21.497067 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.496996 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bc3ad56-5026-4312-85fb-0f40baceeee9-istiod-ca-cert" pod "2bc3ad56-5026-4312-85fb-0f40baceeee9" (UID: "2bc3ad56-5026-4312-85fb-0f40baceeee9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:21.497247 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.497222 4455 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bc3ad56-5026-4312-85fb-0f40baceeee9-istiod-ca-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:22:21.497417 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.497251 4455 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bc3ad56-5026-4312-85fb-0f40baceeee9-istio-data\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:22:21.497417 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.497272 4455 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc3ad56-5026-4312-85fb-0f40baceeee9-workload-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:22:21.497417 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.497288 4455 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc3ad56-5026-4312-85fb-0f40baceeee9-credential-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:22:21.497417 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.497302 4455 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bc3ad56-5026-4312-85fb-0f40baceeee9-workload-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:22:21.499664 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.499628 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bc3ad56-5026-4312-85fb-0f40baceeee9-istio-token" pod "2bc3ad56-5026-4312-85fb-0f40baceeee9" (UID: "2bc3ad56-5026-4312-85fb-0f40baceeee9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:21.502561 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.502535 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bc3ad56-5026-4312-85fb-0f40baceeee9-istio-envoy" pod "2bc3ad56-5026-4312-85fb-0f40baceeee9" (UID: "2bc3ad56-5026-4312-85fb-0f40baceeee9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:21.502960 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.502933 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bc3ad56-5026-4312-85fb-0f40baceeee9-kube-api-access-2v9hz" pod "2bc3ad56-5026-4312-85fb-0f40baceeee9" (UID: "2bc3ad56-5026-4312-85fb-0f40baceeee9"). InnerVolumeSpecName "kube-api-access-2v9hz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:21.510771 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.510744 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bc3ad56-5026-4312-85fb-0f40baceeee9-istio-podinfo" pod "2bc3ad56-5026-4312-85fb-0f40baceeee9" (UID: "2bc3ad56-5026-4312-85fb-0f40baceeee9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:22:21.558279 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.558242 4455 generic.go:356] "Generic (PLEG): container finished" podID="2bc3ad56-5026-4312-85fb-0f40baceeee9" containerID="7673c400619052abf4ce86bec66307cfc7501390202d5aca35f03e2900ce264c" exitCode=0 Oct 30 18:22:21.558482 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.558304 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" Oct 30 18:22:21.558482 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.558348 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" event={"ID":"2bc3ad56-5026-4312-85fb-0f40baceeee9","Type":"ContainerDied","Data":"7673c400619052abf4ce86bec66307cfc7501390202d5aca35f03e2900ce264c"} Oct 30 18:22:21.558482 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.558406 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk" event={"ID":"2bc3ad56-5026-4312-85fb-0f40baceeee9","Type":"ContainerDied","Data":"5f0b02c08442f628f08da7608f824aba2a64d39f2edba41d810bd23a16cea7f4"} Oct 30 18:22:21.558482 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.558430 4455 scope.go:122] "RemoveContainer" containerID="7673c400619052abf4ce86bec66307cfc7501390202d5aca35f03e2900ce264c" Oct 30 18:22:21.577243 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.577225 4455 scope.go:122] "RemoveContainer" containerID="7673c400619052abf4ce86bec66307cfc7501390202d5aca35f03e2900ce264c" Oct 30 18:22:21.577537 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:22:21.577516 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7673c400619052abf4ce86bec66307cfc7501390202d5aca35f03e2900ce264c\": container with ID starting with 7673c400619052abf4ce86bec66307cfc7501390202d5aca35f03e2900ce264c not found: ID does not exist" containerID="7673c400619052abf4ce86bec66307cfc7501390202d5aca35f03e2900ce264c" Oct 30 18:22:21.577642 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.577546 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7673c400619052abf4ce86bec66307cfc7501390202d5aca35f03e2900ce264c"} err="failed to get container status \"7673c400619052abf4ce86bec66307cfc7501390202d5aca35f03e2900ce264c\": rpc error: code = NotFound desc = could not find container \"7673c400619052abf4ce86bec66307cfc7501390202d5aca35f03e2900ce264c\": container with ID starting with 7673c400619052abf4ce86bec66307cfc7501390202d5aca35f03e2900ce264c not found: ID does not exist" Oct 30 18:22:21.599677 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.598894 4455 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bc3ad56-5026-4312-85fb-0f40baceeee9-istio-token\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:22:21.599677 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.598925 4455 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bc3ad56-5026-4312-85fb-0f40baceeee9-istio-podinfo\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:22:21.599677 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.598939 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v9hz\" (UniqueName: \"kubernetes.io/projected/2bc3ad56-5026-4312-85fb-0f40baceeee9-kube-api-access-2v9hz\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:22:21.599677 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.598951 4455 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bc3ad56-5026-4312-85fb-0f40baceeee9-istio-envoy\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:22:21.620497 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.620448 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk"] Oct 30 18:22:21.626022 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.625994 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-ttbkk"] Oct 30 18:22:21.635302 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:21.635276 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bc3ad56-5026-4312-85fb-0f40baceeee9" path="/var/lib/kubelet/pods/2bc3ad56-5026-4312-85fb-0f40baceeee9/volumes" Oct 30 18:22:22.339427 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:22.339398 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-9l4x8_909b1ff6-6d4a-4887-9b59-efe2806ac0e8/payload-processing/0.log" Oct 30 18:22:22.530664 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:22.530636 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-394a31-7d9db95ddb-8xl6n_97e9aed5-3f20-4301-b832-a78b35c9fb71/payload-processing/1.log" Oct 30 18:22:24.982078 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:24.982011 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3be60-cd57b57d7-fnl2f" podUID="e084cf95-22eb-4209-84a6-9c942327cbd7" containerName="payload-pre-processing" containerID="cri-o://93fad3ac3046816909627da35cd9ec88ae9a573d412d400de3539e4fae833d37" gracePeriod=25 Oct 30 18:22:25.280081 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:25.280047 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3be60-cd57b57d7-fnl2f" Oct 30 18:22:25.365391 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:25.365349 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e084cf95-22eb-4209-84a6-9c942327cbd7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e084cf95-22eb-4209-84a6-9c942327cbd7-plugins-config-volume\") pod \"e084cf95-22eb-4209-84a6-9c942327cbd7\" (UID: \"e084cf95-22eb-4209-84a6-9c942327cbd7\") " Oct 30 18:22:25.365595 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:25.365496 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e084cf95-22eb-4209-84a6-9c942327cbd7-kube-api-access-vgp9p\" (UniqueName: \"kubernetes.io/projected/e084cf95-22eb-4209-84a6-9c942327cbd7-kube-api-access-vgp9p\") pod \"e084cf95-22eb-4209-84a6-9c942327cbd7\" (UID: \"e084cf95-22eb-4209-84a6-9c942327cbd7\") " Oct 30 18:22:25.365927 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:25.365893 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e084cf95-22eb-4209-84a6-9c942327cbd7-plugins-config-volume" pod "e084cf95-22eb-4209-84a6-9c942327cbd7" (UID: "e084cf95-22eb-4209-84a6-9c942327cbd7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:25.366260 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:25.366216 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e084cf95-22eb-4209-84a6-9c942327cbd7-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:22:25.368891 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:25.368864 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e084cf95-22eb-4209-84a6-9c942327cbd7-kube-api-access-vgp9p" pod "e084cf95-22eb-4209-84a6-9c942327cbd7" (UID: "e084cf95-22eb-4209-84a6-9c942327cbd7"). InnerVolumeSpecName "kube-api-access-vgp9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:25.468161 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:25.468124 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgp9p\" (UniqueName: \"kubernetes.io/projected/e084cf95-22eb-4209-84a6-9c942327cbd7-kube-api-access-vgp9p\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:22:25.593875 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:25.593786 4455 generic.go:356] "Generic (PLEG): container finished" podID="e084cf95-22eb-4209-84a6-9c942327cbd7" containerID="93fad3ac3046816909627da35cd9ec88ae9a573d412d400de3539e4fae833d37" exitCode=0 Oct 30 18:22:25.593875 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:25.593836 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3be60-cd57b57d7-fnl2f" Oct 30 18:22:25.593875 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:25.593865 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3be60-cd57b57d7-fnl2f" event={"ID":"e084cf95-22eb-4209-84a6-9c942327cbd7","Type":"ContainerDied","Data":"93fad3ac3046816909627da35cd9ec88ae9a573d412d400de3539e4fae833d37"} Oct 30 18:22:25.594112 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:25.593943 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3be60-cd57b57d7-fnl2f" event={"ID":"e084cf95-22eb-4209-84a6-9c942327cbd7","Type":"ContainerDied","Data":"0d38cfcfdf806ee0574e648ceb559b4fd290ddb242499d282457ce7978558521"} Oct 30 18:22:25.594112 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:25.593967 4455 scope.go:122] "RemoveContainer" containerID="93fad3ac3046816909627da35cd9ec88ae9a573d412d400de3539e4fae833d37" Oct 30 18:22:25.604875 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:25.604853 4455 scope.go:122] "RemoveContainer" containerID="93fad3ac3046816909627da35cd9ec88ae9a573d412d400de3539e4fae833d37" Oct 30 18:22:25.605163 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:22:25.605128 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93fad3ac3046816909627da35cd9ec88ae9a573d412d400de3539e4fae833d37\": container with ID starting with 93fad3ac3046816909627da35cd9ec88ae9a573d412d400de3539e4fae833d37 not found: ID does not exist" containerID="93fad3ac3046816909627da35cd9ec88ae9a573d412d400de3539e4fae833d37" Oct 30 18:22:25.605225 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:25.605172 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93fad3ac3046816909627da35cd9ec88ae9a573d412d400de3539e4fae833d37"} err="failed to get container status \"93fad3ac3046816909627da35cd9ec88ae9a573d412d400de3539e4fae833d37\": rpc error: code = NotFound desc = could not find container \"93fad3ac3046816909627da35cd9ec88ae9a573d412d400de3539e4fae833d37\": container with ID starting with 93fad3ac3046816909627da35cd9ec88ae9a573d412d400de3539e4fae833d37 not found: ID does not exist" Oct 30 18:22:25.655303 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:25.655270 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a3be60-cd57b57d7-fnl2f"] Oct 30 18:22:25.670666 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:25.670632 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a3be60-cd57b57d7-fnl2f"] Oct 30 18:22:25.681032 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:25.681000 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc"] Oct 30 18:22:25.681264 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:25.681232 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc" podUID="55217532-8833-4f27-92cf-509c8059409b" containerName="maas-api" containerID="cri-o://6c1a97f14e7ca3198a55c13f916db9c2c8d3154bb32c021bd1f1523bab26bde9" gracePeriod=30 Oct 30 18:22:25.857263 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:25.857172 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-bfb464d74-l4lmt"] Oct 30 18:22:25.857480 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:25.857454 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-bfb464d74-l4lmt" podUID="8e55a9f3-53be-4afd-be10-c7a144bc9835" containerName="payload-processing" containerID="cri-o://65ced1793e29a03c9042e3907866d410a35ed38652f2e10f38c21b12a99aa833" gracePeriod=30 Oct 30 18:22:26.023493 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.023465 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc" Oct 30 18:22:26.076891 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.076862 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55217532-8833-4f27-92cf-509c8059409b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/55217532-8833-4f27-92cf-509c8059409b-metrics-certs\") pod \"55217532-8833-4f27-92cf-509c8059409b\" (UID: \"55217532-8833-4f27-92cf-509c8059409b\") " Oct 30 18:22:26.077049 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.076935 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55217532-8833-4f27-92cf-509c8059409b-kube-api-access-5sr2c\" (UniqueName: \"kubernetes.io/projected/55217532-8833-4f27-92cf-509c8059409b-kube-api-access-5sr2c\") pod \"55217532-8833-4f27-92cf-509c8059409b\" (UID: \"55217532-8833-4f27-92cf-509c8059409b\") " Oct 30 18:22:26.077297 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.077272 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55217532-8833-4f27-92cf-509c8059409b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55217532-8833-4f27-92cf-509c8059409b-maas-api-tls\") pod \"55217532-8833-4f27-92cf-509c8059409b\" (UID: \"55217532-8833-4f27-92cf-509c8059409b\") " Oct 30 18:22:26.077412 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.077379 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55217532-8833-4f27-92cf-509c8059409b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55217532-8833-4f27-92cf-509c8059409b-openshift-service-ca\") pod \"55217532-8833-4f27-92cf-509c8059409b\" (UID: \"55217532-8833-4f27-92cf-509c8059409b\") " Oct 30 18:22:26.078762 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.078732 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55217532-8833-4f27-92cf-509c8059409b-openshift-service-ca" pod "55217532-8833-4f27-92cf-509c8059409b" (UID: "55217532-8833-4f27-92cf-509c8059409b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:26.081151 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.081128 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55217532-8833-4f27-92cf-509c8059409b-kube-api-access-5sr2c" pod "55217532-8833-4f27-92cf-509c8059409b" (UID: "55217532-8833-4f27-92cf-509c8059409b"). InnerVolumeSpecName "kube-api-access-5sr2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:26.090367 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.086159 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55217532-8833-4f27-92cf-509c8059409b-metrics-certs" pod "55217532-8833-4f27-92cf-509c8059409b" (UID: "55217532-8833-4f27-92cf-509c8059409b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:22:26.091360 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.090672 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55217532-8833-4f27-92cf-509c8059409b-maas-api-tls" pod "55217532-8833-4f27-92cf-509c8059409b" (UID: "55217532-8833-4f27-92cf-509c8059409b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:22:26.180895 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.180856 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55217532-8833-4f27-92cf-509c8059409b-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:22:26.180895 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.180887 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55217532-8833-4f27-92cf-509c8059409b-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:22:26.181094 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.180902 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55217532-8833-4f27-92cf-509c8059409b-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:22:26.181094 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.180914 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sr2c\" (UniqueName: \"kubernetes.io/projected/55217532-8833-4f27-92cf-509c8059409b-kube-api-access-5sr2c\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:22:26.186443 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.186410 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-bfb464d74-l4lmt" Oct 30 18:22:26.213968 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.213939 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc"] Oct 30 18:22:26.214186 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.214164 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc" podUID="917f5f39-d463-4c4f-a122-13ed5a3d6075" containerName="maas-api" containerID="cri-o://09794edd3789b2e51595a2193f93670b875b5f38018091d107c0097a2b2c9146" gracePeriod=30 Oct 30 18:22:26.283428 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.283397 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e55a9f3-53be-4afd-be10-c7a144bc9835-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e55a9f3-53be-4afd-be10-c7a144bc9835-plugins-config-volume\") pod \"8e55a9f3-53be-4afd-be10-c7a144bc9835\" (UID: \"8e55a9f3-53be-4afd-be10-c7a144bc9835\") " Oct 30 18:22:26.283643 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.283533 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e55a9f3-53be-4afd-be10-c7a144bc9835-kube-api-access-5f4rf\" (UniqueName: \"kubernetes.io/projected/8e55a9f3-53be-4afd-be10-c7a144bc9835-kube-api-access-5f4rf\") pod \"8e55a9f3-53be-4afd-be10-c7a144bc9835\" (UID: \"8e55a9f3-53be-4afd-be10-c7a144bc9835\") " Oct 30 18:22:26.284519 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.284489 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e55a9f3-53be-4afd-be10-c7a144bc9835-plugins-config-volume" pod "8e55a9f3-53be-4afd-be10-c7a144bc9835" (UID: "8e55a9f3-53be-4afd-be10-c7a144bc9835"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:26.286656 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.286630 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e55a9f3-53be-4afd-be10-c7a144bc9835-kube-api-access-5f4rf" pod "8e55a9f3-53be-4afd-be10-c7a144bc9835" (UID: "8e55a9f3-53be-4afd-be10-c7a144bc9835"). InnerVolumeSpecName "kube-api-access-5f4rf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:26.386544 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.386494 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e55a9f3-53be-4afd-be10-c7a144bc9835-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:22:26.386544 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.386526 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f4rf\" (UniqueName: \"kubernetes.io/projected/8e55a9f3-53be-4afd-be10-c7a144bc9835-kube-api-access-5f4rf\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:22:26.439820 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.438703 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-5fc46c9879-vzjtw"] Oct 30 18:22:26.439820 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.438939 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-5fc46c9879-vzjtw" podUID="cbcc3d28-dd08-4146-91d5-9138a5057230" containerName="payload-pre-processing" containerID="cri-o://4785b788ffa6b3b7f0e2721d0503e49445134b83da1312cebc7d2d1e44a8ff24" gracePeriod=30 Oct 30 18:22:26.553737 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.553707 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc" Oct 30 18:22:26.629301 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.628416 4455 generic.go:356] "Generic (PLEG): container finished" podID="cbcc3d28-dd08-4146-91d5-9138a5057230" containerID="4785b788ffa6b3b7f0e2721d0503e49445134b83da1312cebc7d2d1e44a8ff24" exitCode=0 Oct 30 18:22:26.629301 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.628541 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-5fc46c9879-vzjtw" event={"ID":"cbcc3d28-dd08-4146-91d5-9138a5057230","Type":"ContainerDied","Data":"4785b788ffa6b3b7f0e2721d0503e49445134b83da1312cebc7d2d1e44a8ff24"} Oct 30 18:22:26.630616 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.630582 4455 generic.go:356] "Generic (PLEG): container finished" podID="8e55a9f3-53be-4afd-be10-c7a144bc9835" containerID="65ced1793e29a03c9042e3907866d410a35ed38652f2e10f38c21b12a99aa833" exitCode=0 Oct 30 18:22:26.630806 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.630784 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-bfb464d74-l4lmt" event={"ID":"8e55a9f3-53be-4afd-be10-c7a144bc9835","Type":"ContainerDied","Data":"65ced1793e29a03c9042e3907866d410a35ed38652f2e10f38c21b12a99aa833"} Oct 30 18:22:26.630874 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.630833 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-bfb464d74-l4lmt" event={"ID":"8e55a9f3-53be-4afd-be10-c7a144bc9835","Type":"ContainerDied","Data":"45ca875973d45dd945e4cfc6f932971eefd1123d5610c77a338abd0587aed0da"} Oct 30 18:22:26.630929 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.630901 4455 scope.go:122] "RemoveContainer" containerID="65ced1793e29a03c9042e3907866d410a35ed38652f2e10f38c21b12a99aa833" Oct 30 18:22:26.631035 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.631022 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-bfb464d74-l4lmt" Oct 30 18:22:26.646117 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.646082 4455 generic.go:356] "Generic (PLEG): container finished" podID="55217532-8833-4f27-92cf-509c8059409b" containerID="6c1a97f14e7ca3198a55c13f916db9c2c8d3154bb32c021bd1f1523bab26bde9" exitCode=0 Oct 30 18:22:26.646458 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.646434 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc" event={"ID":"55217532-8833-4f27-92cf-509c8059409b","Type":"ContainerDied","Data":"6c1a97f14e7ca3198a55c13f916db9c2c8d3154bb32c021bd1f1523bab26bde9"} Oct 30 18:22:26.646627 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.646611 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc" event={"ID":"55217532-8833-4f27-92cf-509c8059409b","Type":"ContainerDied","Data":"fa980990e4e6162d18ef6231e9235f3803ee44b2bfa42dc0f228e5d99974d0e1"} Oct 30 18:22:26.646770 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.646467 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc" Oct 30 18:22:26.651878 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.651028 4455 generic.go:356] "Generic (PLEG): container finished" podID="917f5f39-d463-4c4f-a122-13ed5a3d6075" containerID="09794edd3789b2e51595a2193f93670b875b5f38018091d107c0097a2b2c9146" exitCode=0 Oct 30 18:22:26.651878 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.651112 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc" event={"ID":"917f5f39-d463-4c4f-a122-13ed5a3d6075","Type":"ContainerDied","Data":"09794edd3789b2e51595a2193f93670b875b5f38018091d107c0097a2b2c9146"} Oct 30 18:22:26.651878 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.651153 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc" event={"ID":"917f5f39-d463-4c4f-a122-13ed5a3d6075","Type":"ContainerDied","Data":"bafb78641a4b7934e5e9f65f796f4877fab16c0a908e136c14ff0f738aee23cd"} Oct 30 18:22:26.651878 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.651227 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc" Oct 30 18:22:26.657590 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.657572 4455 scope.go:122] "RemoveContainer" containerID="65ced1793e29a03c9042e3907866d410a35ed38652f2e10f38c21b12a99aa833" Oct 30 18:22:26.657934 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:22:26.657914 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65ced1793e29a03c9042e3907866d410a35ed38652f2e10f38c21b12a99aa833\": container with ID starting with 65ced1793e29a03c9042e3907866d410a35ed38652f2e10f38c21b12a99aa833 not found: ID does not exist" containerID="65ced1793e29a03c9042e3907866d410a35ed38652f2e10f38c21b12a99aa833" Oct 30 18:22:26.657987 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.657943 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65ced1793e29a03c9042e3907866d410a35ed38652f2e10f38c21b12a99aa833"} err="failed to get container status \"65ced1793e29a03c9042e3907866d410a35ed38652f2e10f38c21b12a99aa833\": rpc error: code = NotFound desc = could not find container \"65ced1793e29a03c9042e3907866d410a35ed38652f2e10f38c21b12a99aa833\": container with ID starting with 65ced1793e29a03c9042e3907866d410a35ed38652f2e10f38c21b12a99aa833 not found: ID does not exist" Oct 30 18:22:26.657987 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.657961 4455 scope.go:122] "RemoveContainer" containerID="6c1a97f14e7ca3198a55c13f916db9c2c8d3154bb32c021bd1f1523bab26bde9" Oct 30 18:22:26.675779 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.675749 4455 scope.go:122] "RemoveContainer" containerID="6c1a97f14e7ca3198a55c13f916db9c2c8d3154bb32c021bd1f1523bab26bde9" Oct 30 18:22:26.676245 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:22:26.676216 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c1a97f14e7ca3198a55c13f916db9c2c8d3154bb32c021bd1f1523bab26bde9\": container with ID starting with 6c1a97f14e7ca3198a55c13f916db9c2c8d3154bb32c021bd1f1523bab26bde9 not found: ID does not exist" containerID="6c1a97f14e7ca3198a55c13f916db9c2c8d3154bb32c021bd1f1523bab26bde9" Oct 30 18:22:26.676374 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.676253 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c1a97f14e7ca3198a55c13f916db9c2c8d3154bb32c021bd1f1523bab26bde9"} err="failed to get container status \"6c1a97f14e7ca3198a55c13f916db9c2c8d3154bb32c021bd1f1523bab26bde9\": rpc error: code = NotFound desc = could not find container \"6c1a97f14e7ca3198a55c13f916db9c2c8d3154bb32c021bd1f1523bab26bde9\": container with ID starting with 6c1a97f14e7ca3198a55c13f916db9c2c8d3154bb32c021bd1f1523bab26bde9 not found: ID does not exist" Oct 30 18:22:26.676374 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.676282 4455 scope.go:122] "RemoveContainer" containerID="09794edd3789b2e51595a2193f93670b875b5f38018091d107c0097a2b2c9146" Oct 30 18:22:26.688788 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.688767 4455 scope.go:122] "RemoveContainer" containerID="09794edd3789b2e51595a2193f93670b875b5f38018091d107c0097a2b2c9146" Oct 30 18:22:26.689241 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:22:26.689166 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09794edd3789b2e51595a2193f93670b875b5f38018091d107c0097a2b2c9146\": container with ID starting with 09794edd3789b2e51595a2193f93670b875b5f38018091d107c0097a2b2c9146 not found: ID does not exist" containerID="09794edd3789b2e51595a2193f93670b875b5f38018091d107c0097a2b2c9146" Oct 30 18:22:26.689241 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.689205 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09794edd3789b2e51595a2193f93670b875b5f38018091d107c0097a2b2c9146"} err="failed to get container status \"09794edd3789b2e51595a2193f93670b875b5f38018091d107c0097a2b2c9146\": rpc error: code = NotFound desc = could not find container \"09794edd3789b2e51595a2193f93670b875b5f38018091d107c0097a2b2c9146\": container with ID starting with 09794edd3789b2e51595a2193f93670b875b5f38018091d107c0097a2b2c9146 not found: ID does not exist" Oct 30 18:22:26.715703 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.715542 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/917f5f39-d463-4c4f-a122-13ed5a3d6075-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/917f5f39-d463-4c4f-a122-13ed5a3d6075-maas-api-tls\") pod \"917f5f39-d463-4c4f-a122-13ed5a3d6075\" (UID: \"917f5f39-d463-4c4f-a122-13ed5a3d6075\") " Oct 30 18:22:26.715703 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.715633 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/917f5f39-d463-4c4f-a122-13ed5a3d6075-metrics-certs\" (UniqueName: \"kubernetes.io/secret/917f5f39-d463-4c4f-a122-13ed5a3d6075-metrics-certs\") pod \"917f5f39-d463-4c4f-a122-13ed5a3d6075\" (UID: \"917f5f39-d463-4c4f-a122-13ed5a3d6075\") " Oct 30 18:22:26.717512 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.717448 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/917f5f39-d463-4c4f-a122-13ed5a3d6075-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/917f5f39-d463-4c4f-a122-13ed5a3d6075-openshift-service-ca\") pod \"917f5f39-d463-4c4f-a122-13ed5a3d6075\" (UID: \"917f5f39-d463-4c4f-a122-13ed5a3d6075\") " Oct 30 18:22:26.736349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.718034 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/917f5f39-d463-4c4f-a122-13ed5a3d6075-kube-api-access-7ms25\" (UniqueName: \"kubernetes.io/projected/917f5f39-d463-4c4f-a122-13ed5a3d6075-kube-api-access-7ms25\") pod \"917f5f39-d463-4c4f-a122-13ed5a3d6075\" (UID: \"917f5f39-d463-4c4f-a122-13ed5a3d6075\") " Oct 30 18:22:26.747709 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.745933 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-bfb464d74-l4lmt"] Oct 30 18:22:26.750706 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.750260 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8dc70-bfb464d74-l4lmt"] Oct 30 18:22:26.754593 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.754549 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/917f5f39-d463-4c4f-a122-13ed5a3d6075-kube-api-access-7ms25" pod "917f5f39-d463-4c4f-a122-13ed5a3d6075" (UID: "917f5f39-d463-4c4f-a122-13ed5a3d6075"). InnerVolumeSpecName "kube-api-access-7ms25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:26.861447 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.852621 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/917f5f39-d463-4c4f-a122-13ed5a3d6075-metrics-certs" pod "917f5f39-d463-4c4f-a122-13ed5a3d6075" (UID: "917f5f39-d463-4c4f-a122-13ed5a3d6075"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:22:26.871346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.864611 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/917f5f39-d463-4c4f-a122-13ed5a3d6075-openshift-service-ca" pod "917f5f39-d463-4c4f-a122-13ed5a3d6075" (UID: "917f5f39-d463-4c4f-a122-13ed5a3d6075"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:26.871346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.866045 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/917f5f39-d463-4c4f-a122-13ed5a3d6075-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:22:26.871346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.866082 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/917f5f39-d463-4c4f-a122-13ed5a3d6075-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:22:26.871346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.866098 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ms25\" (UniqueName: \"kubernetes.io/projected/917f5f39-d463-4c4f-a122-13ed5a3d6075-kube-api-access-7ms25\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:22:26.965451 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.965397 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc"] Oct 30 18:22:26.981370 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.981277 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/917f5f39-d463-4c4f-a122-13ed5a3d6075-maas-api-tls" pod "917f5f39-d463-4c4f-a122-13ed5a3d6075" (UID: "917f5f39-d463-4c4f-a122-13ed5a3d6075"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:22:26.994038 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:26.994001 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/917f5f39-d463-4c4f-a122-13ed5a3d6075-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:22:27.019097 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:27.015280 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8dc70-6456969f-79zjc"] Oct 30 18:22:27.072599 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:27.072565 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-5fc46c9879-vzjtw" Oct 30 18:22:27.106669 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:27.106632 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc"] Oct 30 18:22:27.110696 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:27.110667 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-069812-55bf6d4f4d-hcbkc"] Oct 30 18:22:27.202375 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:27.202312 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbcc3d28-dd08-4146-91d5-9138a5057230-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbcc3d28-dd08-4146-91d5-9138a5057230-plugins-config-volume\") pod \"cbcc3d28-dd08-4146-91d5-9138a5057230\" (UID: \"cbcc3d28-dd08-4146-91d5-9138a5057230\") " Oct 30 18:22:27.214090 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:27.202688 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbcc3d28-dd08-4146-91d5-9138a5057230-kube-api-access-fvjjn\" (UniqueName: \"kubernetes.io/projected/cbcc3d28-dd08-4146-91d5-9138a5057230-kube-api-access-fvjjn\") pod \"cbcc3d28-dd08-4146-91d5-9138a5057230\" (UID: \"cbcc3d28-dd08-4146-91d5-9138a5057230\") " Oct 30 18:22:27.215714 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:27.215638 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbcc3d28-dd08-4146-91d5-9138a5057230-plugins-config-volume" pod "cbcc3d28-dd08-4146-91d5-9138a5057230" (UID: "cbcc3d28-dd08-4146-91d5-9138a5057230"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:27.225101 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:27.225060 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbcc3d28-dd08-4146-91d5-9138a5057230-kube-api-access-fvjjn" pod "cbcc3d28-dd08-4146-91d5-9138a5057230" (UID: "cbcc3d28-dd08-4146-91d5-9138a5057230"). InnerVolumeSpecName "kube-api-access-fvjjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:27.316960 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:27.316922 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbcc3d28-dd08-4146-91d5-9138a5057230-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:22:27.316960 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:27.316957 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvjjn\" (UniqueName: \"kubernetes.io/projected/cbcc3d28-dd08-4146-91d5-9138a5057230-kube-api-access-fvjjn\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:22:27.636830 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:27.636748 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55217532-8833-4f27-92cf-509c8059409b" path="/var/lib/kubelet/pods/55217532-8833-4f27-92cf-509c8059409b/volumes" Oct 30 18:22:27.637227 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:27.637213 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e55a9f3-53be-4afd-be10-c7a144bc9835" path="/var/lib/kubelet/pods/8e55a9f3-53be-4afd-be10-c7a144bc9835/volumes" Oct 30 18:22:27.637710 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:27.637694 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="917f5f39-d463-4c4f-a122-13ed5a3d6075" path="/var/lib/kubelet/pods/917f5f39-d463-4c4f-a122-13ed5a3d6075/volumes" Oct 30 18:22:27.638462 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:27.638447 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e084cf95-22eb-4209-84a6-9c942327cbd7" path="/var/lib/kubelet/pods/e084cf95-22eb-4209-84a6-9c942327cbd7/volumes" Oct 30 18:22:27.665023 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:27.664970 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-5fc46c9879-vzjtw" event={"ID":"cbcc3d28-dd08-4146-91d5-9138a5057230","Type":"ContainerDied","Data":"58dd9c55b7ac1c219c0d0e5ec45a6a2cdc6ecc31cd9be04b417893dda1441571"} Oct 30 18:22:27.665023 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:27.665004 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-5fc46c9879-vzjtw" Oct 30 18:22:27.665291 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:27.665052 4455 scope.go:122] "RemoveContainer" containerID="4785b788ffa6b3b7f0e2721d0503e49445134b83da1312cebc7d2d1e44a8ff24" Oct 30 18:22:27.732450 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:27.732412 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-5fc46c9879-vzjtw"] Oct 30 18:22:27.739600 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:27.739571 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-069812-5fc46c9879-vzjtw"] Oct 30 18:22:29.640764 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:29.637368 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbcc3d28-dd08-4146-91d5-9138a5057230" path="/var/lib/kubelet/pods/cbcc3d28-dd08-4146-91d5-9138a5057230/volumes" Oct 30 18:22:30.387406 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:22:30.387362 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:22:34.004691 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.004654 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d42172-54484dd44c-nn9fs"] Oct 30 18:22:34.005405 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.005389 4455 cpu_manager.go:414] "containerMap: removing container" podUID="22a3bbc2-f6ee-409d-937b-a0faf2903240" containerName="maas-api" Oct 30 18:22:34.005405 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.005405 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22a3bbc2-f6ee-409d-937b-a0faf2903240" containerName="maas-api" Oct 30 18:22:34.005553 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.005419 4455 cpu_manager.go:414] "containerMap: removing container" podUID="e084cf95-22eb-4209-84a6-9c942327cbd7" containerName="payload-pre-processing" Oct 30 18:22:34.005553 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.005424 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e084cf95-22eb-4209-84a6-9c942327cbd7" containerName="payload-pre-processing" Oct 30 18:22:34.005553 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.005440 4455 cpu_manager.go:414] "containerMap: removing container" podUID="55217532-8833-4f27-92cf-509c8059409b" containerName="maas-api" Oct 30 18:22:34.005553 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.005445 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55217532-8833-4f27-92cf-509c8059409b" containerName="maas-api" Oct 30 18:22:34.005553 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.005464 4455 cpu_manager.go:414] "containerMap: removing container" podUID="917f5f39-d463-4c4f-a122-13ed5a3d6075" containerName="maas-api" Oct 30 18:22:34.005553 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.005470 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="917f5f39-d463-4c4f-a122-13ed5a3d6075" containerName="maas-api" Oct 30 18:22:34.005553 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.005480 4455 cpu_manager.go:414] "containerMap: removing container" podUID="8e55a9f3-53be-4afd-be10-c7a144bc9835" containerName="payload-processing" Oct 30 18:22:34.005553 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.005484 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e55a9f3-53be-4afd-be10-c7a144bc9835" containerName="payload-processing" Oct 30 18:22:34.005553 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.005493 4455 cpu_manager.go:414] "containerMap: removing container" podUID="cbcc3d28-dd08-4146-91d5-9138a5057230" containerName="payload-pre-processing" Oct 30 18:22:34.005553 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.005497 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbcc3d28-dd08-4146-91d5-9138a5057230" containerName="payload-pre-processing" Oct 30 18:22:34.005553 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.005505 4455 cpu_manager.go:414] "containerMap: removing container" podUID="f4cf887a-fbe0-4738-b919-5110b2f77a7b" containerName="maas-api" Oct 30 18:22:34.005553 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.005509 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4cf887a-fbe0-4738-b919-5110b2f77a7b" containerName="maas-api" Oct 30 18:22:34.005553 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.005518 4455 cpu_manager.go:414] "containerMap: removing container" podUID="2bc3ad56-5026-4312-85fb-0f40baceeee9" containerName="istio-proxy" Oct 30 18:22:34.005553 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.005522 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bc3ad56-5026-4312-85fb-0f40baceeee9" containerName="istio-proxy" Oct 30 18:22:34.006231 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.005625 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="917f5f39-d463-4c4f-a122-13ed5a3d6075" containerName="maas-api" Oct 30 18:22:34.006231 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.005636 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e55a9f3-53be-4afd-be10-c7a144bc9835" containerName="payload-processing" Oct 30 18:22:34.006231 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.005641 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbcc3d28-dd08-4146-91d5-9138a5057230" containerName="payload-pre-processing" Oct 30 18:22:34.006231 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.005646 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4cf887a-fbe0-4738-b919-5110b2f77a7b" containerName="maas-api" Oct 30 18:22:34.006231 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.005653 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bc3ad56-5026-4312-85fb-0f40baceeee9" containerName="istio-proxy" Oct 30 18:22:34.006231 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.005663 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="22a3bbc2-f6ee-409d-937b-a0faf2903240" containerName="maas-api" Oct 30 18:22:34.006231 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.005669 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="e084cf95-22eb-4209-84a6-9c942327cbd7" containerName="payload-pre-processing" Oct 30 18:22:34.006231 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.005674 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="55217532-8833-4f27-92cf-509c8059409b" containerName="maas-api" Oct 30 18:22:34.011611 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.011584 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d42172-54484dd44c-nn9fs" Oct 30 18:22:34.031433 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.031402 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d42172-54484dd44c-nn9fs"] Oct 30 18:22:34.040531 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.040498 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-d42172\"" Oct 30 18:22:34.044285 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.044261 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-d42172-dockercfg-fdp2l\"" Oct 30 18:22:34.240013 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.239728 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb23ee45-fc23-4de0-9e81-71e087c0c502-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d42172-54484dd44c-nn9fs\" (UID: \"eb23ee45-fc23-4de0-9e81-71e087c0c502\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d42172-54484dd44c-nn9fs" Oct 30 18:22:34.240013 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.239870 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mwdv\" (UniqueName: \"kubernetes.io/projected/eb23ee45-fc23-4de0-9e81-71e087c0c502-kube-api-access-5mwdv\") pod \"payload-pre-processing-e2e-praxis-a-d42172-54484dd44c-nn9fs\" (UID: \"eb23ee45-fc23-4de0-9e81-71e087c0c502\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d42172-54484dd44c-nn9fs" Oct 30 18:22:34.342873 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.342789 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb23ee45-fc23-4de0-9e81-71e087c0c502-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d42172-54484dd44c-nn9fs\" (UID: \"eb23ee45-fc23-4de0-9e81-71e087c0c502\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d42172-54484dd44c-nn9fs" Oct 30 18:22:34.343225 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.343197 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mwdv\" (UniqueName: \"kubernetes.io/projected/eb23ee45-fc23-4de0-9e81-71e087c0c502-kube-api-access-5mwdv\") pod \"payload-pre-processing-e2e-praxis-a-d42172-54484dd44c-nn9fs\" (UID: \"eb23ee45-fc23-4de0-9e81-71e087c0c502\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d42172-54484dd44c-nn9fs" Oct 30 18:22:34.344753 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.344733 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb23ee45-fc23-4de0-9e81-71e087c0c502-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d42172-54484dd44c-nn9fs\" (UID: \"eb23ee45-fc23-4de0-9e81-71e087c0c502\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d42172-54484dd44c-nn9fs" Oct 30 18:22:34.370556 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.369828 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mwdv\" (UniqueName: \"kubernetes.io/projected/eb23ee45-fc23-4de0-9e81-71e087c0c502-kube-api-access-5mwdv\") pod \"payload-pre-processing-e2e-praxis-a-d42172-54484dd44c-nn9fs\" (UID: \"eb23ee45-fc23-4de0-9e81-71e087c0c502\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d42172-54484dd44c-nn9fs" Oct 30 18:22:34.630373 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:34.630230 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d42172-54484dd44c-nn9fs" Oct 30 18:22:35.063510 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:35.061457 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d42172-54484dd44c-nn9fs"] Oct 30 18:22:35.072070 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:22:35.072036 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb23ee45_fc23_4de0_9e81_71e087c0c502.slice/crio-28a93fac0edb6407416b017b5befbc481c74431dd4b1b0e486cb0cdcd8170871 WatchSource:0}: Error finding container 28a93fac0edb6407416b017b5befbc481c74431dd4b1b0e486cb0cdcd8170871: Status 404 returned error can't find the container with id 28a93fac0edb6407416b017b5befbc481c74431dd4b1b0e486cb0cdcd8170871 Oct 30 18:22:35.821199 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:35.821159 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d42172-54484dd44c-nn9fs" event={"ID":"eb23ee45-fc23-4de0-9e81-71e087c0c502","Type":"ContainerStarted","Data":"23f349c98469c940bfc8bf68c577fa543e7e246cd650a07d5a1f9eb0d48d6155"} Oct 30 18:22:35.821414 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:35.821217 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d42172-54484dd44c-nn9fs" event={"ID":"eb23ee45-fc23-4de0-9e81-71e087c0c502","Type":"ContainerStarted","Data":"28a93fac0edb6407416b017b5befbc481c74431dd4b1b0e486cb0cdcd8170871"} Oct 30 18:22:35.821414 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:35.821378 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d42172-54484dd44c-nn9fs" Oct 30 18:22:35.867422 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:35.867362 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d42172-54484dd44c-nn9fs" podStartSLOduration=2.867343234 podStartE2EDuration="2.867343234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:35.864785322 +0000 UTC m=+1808.352422356" watchObservedRunningTime="2026-10-30 18:22:35.867343234 +0000 UTC m=+1808.354980359" Oct 30 18:22:42.840007 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:42.839966 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t"] Oct 30 18:22:42.840563 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:42.840230 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t" podUID="7ef482c4-f1c6-4b06-8e14-220e622fa711" containerName="maas-api" containerID="cri-o://225f3f9713f2f7869a356bf7a3b3756333ace26f90d011996b9a284cb85ffbf2" gracePeriod=30 Oct 30 18:22:43.250398 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.250360 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ca5f33-546df68d7-nnnlz"] Oct 30 18:22:43.292192 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.292155 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ca5f33-546df68d7-nnnlz"] Oct 30 18:22:43.292566 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.292301 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca5f33-546df68d7-nnnlz" Oct 30 18:22:43.328272 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.328135 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-ca5f33-dockercfg-plfmq\"" Oct 30 18:22:43.328272 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.328235 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-ca5f33\"" Oct 30 18:22:43.344211 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.344160 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ca5f33-54dcbfd498-fvclr"] Oct 30 18:22:43.352605 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.352579 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca5f33-54dcbfd498-fvclr" Oct 30 18:22:43.449404 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.448247 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t" Oct 30 18:22:43.452108 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.450462 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzct7\" (UniqueName: \"kubernetes.io/projected/8a49c1e9-b602-4d17-828c-a95ea3325c66-kube-api-access-bzct7\") pod \"payload-processing-e2e-praxis-a-ca5f33-54dcbfd498-fvclr\" (UID: \"8a49c1e9-b602-4d17-828c-a95ea3325c66\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ca5f33-54dcbfd498-fvclr" Oct 30 18:22:43.452108 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.450569 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fdeba52-780c-4631-aded-74a165c8c700-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ca5f33-546df68d7-nnnlz\" (UID: \"0fdeba52-780c-4631-aded-74a165c8c700\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca5f33-546df68d7-nnnlz" Oct 30 18:22:43.452108 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.450712 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a49c1e9-b602-4d17-828c-a95ea3325c66-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ca5f33-54dcbfd498-fvclr\" (UID: \"8a49c1e9-b602-4d17-828c-a95ea3325c66\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ca5f33-54dcbfd498-fvclr" Oct 30 18:22:43.452108 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.450802 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp6nc\" (UniqueName: \"kubernetes.io/projected/0fdeba52-780c-4631-aded-74a165c8c700-kube-api-access-qp6nc\") pod \"payload-pre-processing-e2e-praxis-a-ca5f33-546df68d7-nnnlz\" (UID: \"0fdeba52-780c-4631-aded-74a165c8c700\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca5f33-546df68d7-nnnlz" Oct 30 18:22:43.557466 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.557426 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ca5f33-54dcbfd498-fvclr"] Oct 30 18:22:43.576343 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.558306 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ef482c4-f1c6-4b06-8e14-220e622fa711-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ef482c4-f1c6-4b06-8e14-220e622fa711-maas-api-tls\") pod \"7ef482c4-f1c6-4b06-8e14-220e622fa711\" (UID: \"7ef482c4-f1c6-4b06-8e14-220e622fa711\") " Oct 30 18:22:43.576343 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.558430 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ef482c4-f1c6-4b06-8e14-220e622fa711-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ef482c4-f1c6-4b06-8e14-220e622fa711-openshift-service-ca\") pod \"7ef482c4-f1c6-4b06-8e14-220e622fa711\" (UID: \"7ef482c4-f1c6-4b06-8e14-220e622fa711\") " Oct 30 18:22:43.576343 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.558537 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ef482c4-f1c6-4b06-8e14-220e622fa711-kube-api-access-8lpsm\" (UniqueName: \"kubernetes.io/projected/7ef482c4-f1c6-4b06-8e14-220e622fa711-kube-api-access-8lpsm\") pod \"7ef482c4-f1c6-4b06-8e14-220e622fa711\" (UID: \"7ef482c4-f1c6-4b06-8e14-220e622fa711\") " Oct 30 18:22:43.576343 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.558591 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ef482c4-f1c6-4b06-8e14-220e622fa711-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ef482c4-f1c6-4b06-8e14-220e622fa711-metrics-certs\") pod \"7ef482c4-f1c6-4b06-8e14-220e622fa711\" (UID: \"7ef482c4-f1c6-4b06-8e14-220e622fa711\") " Oct 30 18:22:43.576343 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.559013 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzct7\" (UniqueName: \"kubernetes.io/projected/8a49c1e9-b602-4d17-828c-a95ea3325c66-kube-api-access-bzct7\") pod \"payload-processing-e2e-praxis-a-ca5f33-54dcbfd498-fvclr\" (UID: \"8a49c1e9-b602-4d17-828c-a95ea3325c66\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ca5f33-54dcbfd498-fvclr" Oct 30 18:22:43.576343 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.559117 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fdeba52-780c-4631-aded-74a165c8c700-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ca5f33-546df68d7-nnnlz\" (UID: \"0fdeba52-780c-4631-aded-74a165c8c700\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca5f33-546df68d7-nnnlz" Oct 30 18:22:43.576343 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.559248 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a49c1e9-b602-4d17-828c-a95ea3325c66-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ca5f33-54dcbfd498-fvclr\" (UID: \"8a49c1e9-b602-4d17-828c-a95ea3325c66\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ca5f33-54dcbfd498-fvclr" Oct 30 18:22:43.576343 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.559399 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp6nc\" (UniqueName: \"kubernetes.io/projected/0fdeba52-780c-4631-aded-74a165c8c700-kube-api-access-qp6nc\") pod \"payload-pre-processing-e2e-praxis-a-ca5f33-546df68d7-nnnlz\" (UID: \"0fdeba52-780c-4631-aded-74a165c8c700\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca5f33-546df68d7-nnnlz" Oct 30 18:22:43.579642 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.579610 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fdeba52-780c-4631-aded-74a165c8c700-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ca5f33-546df68d7-nnnlz\" (UID: \"0fdeba52-780c-4631-aded-74a165c8c700\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca5f33-546df68d7-nnnlz" Oct 30 18:22:43.583529 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.580894 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ef482c4-f1c6-4b06-8e14-220e622fa711-openshift-service-ca" pod "7ef482c4-f1c6-4b06-8e14-220e622fa711" (UID: "7ef482c4-f1c6-4b06-8e14-220e622fa711"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:43.583737 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.582163 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a49c1e9-b602-4d17-828c-a95ea3325c66-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ca5f33-54dcbfd498-fvclr\" (UID: \"8a49c1e9-b602-4d17-828c-a95ea3325c66\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ca5f33-54dcbfd498-fvclr" Oct 30 18:22:43.606168 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.605845 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ef482c4-f1c6-4b06-8e14-220e622fa711-metrics-certs" pod "7ef482c4-f1c6-4b06-8e14-220e622fa711" (UID: "7ef482c4-f1c6-4b06-8e14-220e622fa711"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:22:43.607552 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.607214 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ef482c4-f1c6-4b06-8e14-220e622fa711-kube-api-access-8lpsm" pod "7ef482c4-f1c6-4b06-8e14-220e622fa711" (UID: "7ef482c4-f1c6-4b06-8e14-220e622fa711"). InnerVolumeSpecName "kube-api-access-8lpsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:43.609860 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.609820 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ef482c4-f1c6-4b06-8e14-220e622fa711-maas-api-tls" pod "7ef482c4-f1c6-4b06-8e14-220e622fa711" (UID: "7ef482c4-f1c6-4b06-8e14-220e622fa711"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:22:43.611593 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.611520 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzct7\" (UniqueName: \"kubernetes.io/projected/8a49c1e9-b602-4d17-828c-a95ea3325c66-kube-api-access-bzct7\") pod \"payload-processing-e2e-praxis-a-ca5f33-54dcbfd498-fvclr\" (UID: \"8a49c1e9-b602-4d17-828c-a95ea3325c66\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ca5f33-54dcbfd498-fvclr" Oct 30 18:22:43.616148 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.616032 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp6nc\" (UniqueName: \"kubernetes.io/projected/0fdeba52-780c-4631-aded-74a165c8c700-kube-api-access-qp6nc\") pod \"payload-pre-processing-e2e-praxis-a-ca5f33-546df68d7-nnnlz\" (UID: \"0fdeba52-780c-4631-aded-74a165c8c700\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca5f33-546df68d7-nnnlz" Oct 30 18:22:43.658150 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.658072 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca5f33-546df68d7-nnnlz" Oct 30 18:22:43.686799 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.686710 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ef482c4-f1c6-4b06-8e14-220e622fa711-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:22:43.686799 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.686743 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ef482c4-f1c6-4b06-8e14-220e622fa711-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:22:43.686799 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.686758 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lpsm\" (UniqueName: \"kubernetes.io/projected/7ef482c4-f1c6-4b06-8e14-220e622fa711-kube-api-access-8lpsm\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:22:43.686799 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.686771 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ef482c4-f1c6-4b06-8e14-220e622fa711-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:22:43.771312 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.771273 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca5f33-54dcbfd498-fvclr" Oct 30 18:22:43.907429 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.907394 4455 generic.go:356] "Generic (PLEG): container finished" podID="7ef482c4-f1c6-4b06-8e14-220e622fa711" containerID="225f3f9713f2f7869a356bf7a3b3756333ace26f90d011996b9a284cb85ffbf2" exitCode=0 Oct 30 18:22:43.907903 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.907642 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t" event={"ID":"7ef482c4-f1c6-4b06-8e14-220e622fa711","Type":"ContainerDied","Data":"225f3f9713f2f7869a356bf7a3b3756333ace26f90d011996b9a284cb85ffbf2"} Oct 30 18:22:43.907903 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.907701 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t" event={"ID":"7ef482c4-f1c6-4b06-8e14-220e622fa711","Type":"ContainerDied","Data":"75c98c72c8bf7e74fda0c2f73c7257185428d29f15e6b52af09074c704f93793"} Oct 30 18:22:43.907903 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.907736 4455 scope.go:122] "RemoveContainer" containerID="225f3f9713f2f7869a356bf7a3b3756333ace26f90d011996b9a284cb85ffbf2" Oct 30 18:22:43.908056 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.907923 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t" Oct 30 18:22:43.928869 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.928842 4455 scope.go:122] "RemoveContainer" containerID="4cfa2179caa134986e43e6ed7da366478a9e6e18926dd1b050bf9c8fc5e8b6df" Oct 30 18:22:43.944886 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.944859 4455 scope.go:122] "RemoveContainer" containerID="225f3f9713f2f7869a356bf7a3b3756333ace26f90d011996b9a284cb85ffbf2" Oct 30 18:22:43.945210 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:22:43.945187 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"225f3f9713f2f7869a356bf7a3b3756333ace26f90d011996b9a284cb85ffbf2\": container with ID starting with 225f3f9713f2f7869a356bf7a3b3756333ace26f90d011996b9a284cb85ffbf2 not found: ID does not exist" containerID="225f3f9713f2f7869a356bf7a3b3756333ace26f90d011996b9a284cb85ffbf2" Oct 30 18:22:43.945309 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.945219 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"225f3f9713f2f7869a356bf7a3b3756333ace26f90d011996b9a284cb85ffbf2"} err="failed to get container status \"225f3f9713f2f7869a356bf7a3b3756333ace26f90d011996b9a284cb85ffbf2\": rpc error: code = NotFound desc = could not find container \"225f3f9713f2f7869a356bf7a3b3756333ace26f90d011996b9a284cb85ffbf2\": container with ID starting with 225f3f9713f2f7869a356bf7a3b3756333ace26f90d011996b9a284cb85ffbf2 not found: ID does not exist" Oct 30 18:22:43.945309 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.945238 4455 scope.go:122] "RemoveContainer" containerID="4cfa2179caa134986e43e6ed7da366478a9e6e18926dd1b050bf9c8fc5e8b6df" Oct 30 18:22:43.945600 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:22:43.945568 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cfa2179caa134986e43e6ed7da366478a9e6e18926dd1b050bf9c8fc5e8b6df\": container with ID starting with 4cfa2179caa134986e43e6ed7da366478a9e6e18926dd1b050bf9c8fc5e8b6df not found: ID does not exist" containerID="4cfa2179caa134986e43e6ed7da366478a9e6e18926dd1b050bf9c8fc5e8b6df" Oct 30 18:22:43.945673 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.945611 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cfa2179caa134986e43e6ed7da366478a9e6e18926dd1b050bf9c8fc5e8b6df"} err="failed to get container status \"4cfa2179caa134986e43e6ed7da366478a9e6e18926dd1b050bf9c8fc5e8b6df\": rpc error: code = NotFound desc = could not find container \"4cfa2179caa134986e43e6ed7da366478a9e6e18926dd1b050bf9c8fc5e8b6df\": container with ID starting with 4cfa2179caa134986e43e6ed7da366478a9e6e18926dd1b050bf9c8fc5e8b6df not found: ID does not exist" Oct 30 18:22:43.970068 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.970029 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t"] Oct 30 18:22:43.986999 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:43.986930 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a3be60-6558d4fd5c-9rq4t"] Oct 30 18:22:44.123678 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:44.123025 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ca5f33-546df68d7-nnnlz"] Oct 30 18:22:44.126497 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:22:44.126469 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fdeba52_780c_4631_aded_74a165c8c700.slice/crio-87cc4ca087d4a94305fede38ac4615738533055b34e2b67169676ef9628e622a WatchSource:0}: Error finding container 87cc4ca087d4a94305fede38ac4615738533055b34e2b67169676ef9628e622a: Status 404 returned error can't find the container with id 87cc4ca087d4a94305fede38ac4615738533055b34e2b67169676ef9628e622a Oct 30 18:22:44.230532 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:22:44.230506 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a49c1e9_b602_4d17_828c_a95ea3325c66.slice/crio-376138a0a8d46553898610979e1fd95ed07bdae7feeb6f7e9aa5942ae985327c WatchSource:0}: Error finding container 376138a0a8d46553898610979e1fd95ed07bdae7feeb6f7e9aa5942ae985327c: Status 404 returned error can't find the container with id 376138a0a8d46553898610979e1fd95ed07bdae7feeb6f7e9aa5942ae985327c Oct 30 18:22:44.233562 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:44.233534 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ca5f33-54dcbfd498-fvclr"] Oct 30 18:22:44.919987 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:44.919949 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca5f33-546df68d7-nnnlz" event={"ID":"0fdeba52-780c-4631-aded-74a165c8c700","Type":"ContainerStarted","Data":"59583ad94991e7462c807ce161969310c06f45dd69992cdb6dd28e7697f843f2"} Oct 30 18:22:44.920513 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:44.920014 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca5f33-546df68d7-nnnlz" event={"ID":"0fdeba52-780c-4631-aded-74a165c8c700","Type":"ContainerStarted","Data":"87cc4ca087d4a94305fede38ac4615738533055b34e2b67169676ef9628e622a"} Oct 30 18:22:44.920513 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:44.920045 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca5f33-546df68d7-nnnlz" Oct 30 18:22:44.921661 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:44.921624 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca5f33-54dcbfd498-fvclr" event={"ID":"8a49c1e9-b602-4d17-828c-a95ea3325c66","Type":"ContainerStarted","Data":"a8634a97c74738f82182a3706e5f6552b14d1f7d1ecb1b4046458f83c3a1d722"} Oct 30 18:22:44.921661 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:44.921662 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca5f33-54dcbfd498-fvclr" event={"ID":"8a49c1e9-b602-4d17-828c-a95ea3325c66","Type":"ContainerStarted","Data":"376138a0a8d46553898610979e1fd95ed07bdae7feeb6f7e9aa5942ae985327c"} Oct 30 18:22:44.921858 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:44.921707 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca5f33-54dcbfd498-fvclr" Oct 30 18:22:44.966746 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:44.966684 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca5f33-546df68d7-nnnlz" podStartSLOduration=1.966663493 podStartE2EDuration="1.966663493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:44.963719524 +0000 UTC m=+1817.451356562" watchObservedRunningTime="2026-10-30 18:22:44.966663493 +0000 UTC m=+1817.454300524" Oct 30 18:22:45.010383 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:45.010293 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca5f33-54dcbfd498-fvclr" podStartSLOduration=2.010278993 podStartE2EDuration="2.010278993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:45.009571105 +0000 UTC m=+1817.497208145" watchObservedRunningTime="2026-10-30 18:22:45.010278993 +0000 UTC m=+1817.497916030" Oct 30 18:22:45.275930 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:45.275889 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ef208a-9ccb8d8f5-jqhpw"] Oct 30 18:22:45.277034 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:45.277003 4455 cpu_manager.go:414] "containerMap: removing container" podUID="7ef482c4-f1c6-4b06-8e14-220e622fa711" containerName="maas-api" Oct 30 18:22:45.277034 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:45.277032 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ef482c4-f1c6-4b06-8e14-220e622fa711" containerName="maas-api" Oct 30 18:22:45.277234 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:45.277063 4455 cpu_manager.go:414] "containerMap: removing container" podUID="7ef482c4-f1c6-4b06-8e14-220e622fa711" containerName="maas-api" Oct 30 18:22:45.277234 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:45.277073 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ef482c4-f1c6-4b06-8e14-220e622fa711" containerName="maas-api" Oct 30 18:22:45.277652 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:45.277631 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ef482c4-f1c6-4b06-8e14-220e622fa711" containerName="maas-api" Oct 30 18:22:45.277733 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:45.277664 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ef482c4-f1c6-4b06-8e14-220e622fa711" containerName="maas-api" Oct 30 18:22:45.282299 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:45.282278 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ef208a-9ccb8d8f5-jqhpw" Oct 30 18:22:45.299179 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:45.299153 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-ef208a\"" Oct 30 18:22:45.299566 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:45.299548 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-ef208a-dockercfg-kqwth\"" Oct 30 18:22:45.304847 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:45.304819 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ef208a-9ccb8d8f5-jqhpw"] Oct 30 18:22:45.445349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:45.431251 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad56482-fee8-47da-85c7-b870fa1e1aeb-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ef208a-9ccb8d8f5-jqhpw\" (UID: \"0ad56482-fee8-47da-85c7-b870fa1e1aeb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ef208a-9ccb8d8f5-jqhpw" Oct 30 18:22:45.445349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:45.431449 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wff5v\" (UniqueName: \"kubernetes.io/projected/0ad56482-fee8-47da-85c7-b870fa1e1aeb-kube-api-access-wff5v\") pod \"payload-pre-processing-e2e-praxis-a-ef208a-9ccb8d8f5-jqhpw\" (UID: \"0ad56482-fee8-47da-85c7-b870fa1e1aeb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ef208a-9ccb8d8f5-jqhpw" Oct 30 18:22:45.500495 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:45.467818 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ef208a-6c5cd4b4d-wrtmz"] Oct 30 18:22:45.504393 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:45.504187 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ef208a-6c5cd4b4d-wrtmz" Oct 30 18:22:45.553396 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:45.545190 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad56482-fee8-47da-85c7-b870fa1e1aeb-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ef208a-9ccb8d8f5-jqhpw\" (UID: \"0ad56482-fee8-47da-85c7-b870fa1e1aeb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ef208a-9ccb8d8f5-jqhpw" Oct 30 18:22:45.632512 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:45.632444 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad56482-fee8-47da-85c7-b870fa1e1aeb-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ef208a-9ccb8d8f5-jqhpw\" (UID: \"0ad56482-fee8-47da-85c7-b870fa1e1aeb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ef208a-9ccb8d8f5-jqhpw" Oct 30 18:22:45.632995 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:45.632966 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wff5v\" (UniqueName: \"kubernetes.io/projected/0ad56482-fee8-47da-85c7-b870fa1e1aeb-kube-api-access-wff5v\") pod \"payload-pre-processing-e2e-praxis-a-ef208a-9ccb8d8f5-jqhpw\" (UID: \"0ad56482-fee8-47da-85c7-b870fa1e1aeb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ef208a-9ccb8d8f5-jqhpw" Oct 30 18:22:45.677650 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:45.677610 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ef482c4-f1c6-4b06-8e14-220e622fa711" path="/var/lib/kubelet/pods/7ef482c4-f1c6-4b06-8e14-220e622fa711/volumes" Oct 30 18:22:45.678104 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:45.678081 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ef208a-6c5cd4b4d-wrtmz"] Oct 30 18:22:45.705972 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:45.700622 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wff5v\" (UniqueName: \"kubernetes.io/projected/0ad56482-fee8-47da-85c7-b870fa1e1aeb-kube-api-access-wff5v\") pod \"payload-pre-processing-e2e-praxis-a-ef208a-9ccb8d8f5-jqhpw\" (UID: \"0ad56482-fee8-47da-85c7-b870fa1e1aeb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ef208a-9ccb8d8f5-jqhpw" Oct 30 18:22:45.735289 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:45.735261 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f7f060-dc48-4a8d-87a1-a760d0351fb6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ef208a-6c5cd4b4d-wrtmz\" (UID: \"58f7f060-dc48-4a8d-87a1-a760d0351fb6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ef208a-6c5cd4b4d-wrtmz" Oct 30 18:22:45.735494 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:45.735441 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7799m\" (UniqueName: \"kubernetes.io/projected/58f7f060-dc48-4a8d-87a1-a760d0351fb6-kube-api-access-7799m\") pod \"payload-processing-e2e-praxis-a-ef208a-6c5cd4b4d-wrtmz\" (UID: \"58f7f060-dc48-4a8d-87a1-a760d0351fb6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ef208a-6c5cd4b4d-wrtmz" Oct 30 18:22:45.836606 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:45.836516 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f7f060-dc48-4a8d-87a1-a760d0351fb6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ef208a-6c5cd4b4d-wrtmz\" (UID: \"58f7f060-dc48-4a8d-87a1-a760d0351fb6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ef208a-6c5cd4b4d-wrtmz" Oct 30 18:22:45.836783 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:45.836641 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7799m\" (UniqueName: \"kubernetes.io/projected/58f7f060-dc48-4a8d-87a1-a760d0351fb6-kube-api-access-7799m\") pod \"payload-processing-e2e-praxis-a-ef208a-6c5cd4b4d-wrtmz\" (UID: \"58f7f060-dc48-4a8d-87a1-a760d0351fb6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ef208a-6c5cd4b4d-wrtmz" Oct 30 18:22:45.837298 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:45.837275 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f7f060-dc48-4a8d-87a1-a760d0351fb6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ef208a-6c5cd4b4d-wrtmz\" (UID: \"58f7f060-dc48-4a8d-87a1-a760d0351fb6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ef208a-6c5cd4b4d-wrtmz" Oct 30 18:22:45.861590 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:45.861564 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7799m\" (UniqueName: \"kubernetes.io/projected/58f7f060-dc48-4a8d-87a1-a760d0351fb6-kube-api-access-7799m\") pod \"payload-processing-e2e-praxis-a-ef208a-6c5cd4b4d-wrtmz\" (UID: \"58f7f060-dc48-4a8d-87a1-a760d0351fb6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ef208a-6c5cd4b4d-wrtmz" Oct 30 18:22:45.906598 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:45.906561 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ef208a-9ccb8d8f5-jqhpw" Oct 30 18:22:45.961594 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:45.961553 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ef208a-6c5cd4b4d-wrtmz" Oct 30 18:22:46.250705 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:46.250152 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ef208a-9ccb8d8f5-jqhpw"] Oct 30 18:22:46.252954 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:22:46.252908 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad56482_fee8_47da_85c7_b870fa1e1aeb.slice/crio-e3027dde22628524a991977fd7e4ae18fb30a2349127f2f360cb26f06b85f795 WatchSource:0}: Error finding container e3027dde22628524a991977fd7e4ae18fb30a2349127f2f360cb26f06b85f795: Status 404 returned error can't find the container with id e3027dde22628524a991977fd7e4ae18fb30a2349127f2f360cb26f06b85f795 Oct 30 18:22:46.326064 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:22:46.325968 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58f7f060_dc48_4a8d_87a1_a760d0351fb6.slice/crio-1b859ef208a9a647b5f028757ea739d954d8489a5784bf4f90115e7ff30d783c WatchSource:0}: Error finding container 1b859ef208a9a647b5f028757ea739d954d8489a5784bf4f90115e7ff30d783c: Status 404 returned error can't find the container with id 1b859ef208a9a647b5f028757ea739d954d8489a5784bf4f90115e7ff30d783c Oct 30 18:22:46.326176 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:46.326104 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ef208a-6c5cd4b4d-wrtmz"] Oct 30 18:22:46.831781 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:46.831751 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d42172-54484dd44c-nn9fs" Oct 30 18:22:46.939420 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:46.939383 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ef208a-9ccb8d8f5-jqhpw" event={"ID":"0ad56482-fee8-47da-85c7-b870fa1e1aeb","Type":"ContainerStarted","Data":"911b93172fae9dcc7a6bfdd099914e3a09d8197df2c59e936cfc451ec1e10855"} Oct 30 18:22:46.939591 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:46.939436 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ef208a-9ccb8d8f5-jqhpw" event={"ID":"0ad56482-fee8-47da-85c7-b870fa1e1aeb","Type":"ContainerStarted","Data":"e3027dde22628524a991977fd7e4ae18fb30a2349127f2f360cb26f06b85f795"} Oct 30 18:22:46.939591 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:46.939468 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ef208a-9ccb8d8f5-jqhpw" Oct 30 18:22:46.941462 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:46.941434 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ef208a-6c5cd4b4d-wrtmz" event={"ID":"58f7f060-dc48-4a8d-87a1-a760d0351fb6","Type":"ContainerStarted","Data":"c31606582efa2941bf89b88cd39ad46df02553998e98883f0e73acbbadd60568"} Oct 30 18:22:46.941551 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:46.941491 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-ef208a-6c5cd4b4d-wrtmz" Oct 30 18:22:46.941551 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:46.941516 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ef208a-6c5cd4b4d-wrtmz" event={"ID":"58f7f060-dc48-4a8d-87a1-a760d0351fb6","Type":"ContainerStarted","Data":"1b859ef208a9a647b5f028757ea739d954d8489a5784bf4f90115e7ff30d783c"} Oct 30 18:22:46.984425 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:46.984368 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ef208a-9ccb8d8f5-jqhpw" podStartSLOduration=1.9843510229999999 podStartE2EDuration="1.984351023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:46.983560752 +0000 UTC m=+1819.471197788" watchObservedRunningTime="2026-10-30 18:22:46.984351023 +0000 UTC m=+1819.471988058" Oct 30 18:22:47.032407 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:47.032346 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-ef208a-6c5cd4b4d-wrtmz" podStartSLOduration=2.032310873 podStartE2EDuration="2.032310873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:47.027992115 +0000 UTC m=+1819.515629150" watchObservedRunningTime="2026-10-30 18:22:47.032310873 +0000 UTC m=+1819.519947908" Oct 30 18:22:49.709044 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:49.709008 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch"] Oct 30 18:22:49.713735 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:49.713716 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch" Oct 30 18:22:49.730977 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:49.730947 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-d42172-serving-cert\"" Oct 30 18:22:49.740514 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:49.740486 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch"] Oct 30 18:22:49.828413 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:49.828377 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00eedd3e-a830-4069-a3fd-4c43c149b943-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch\" (UID: \"00eedd3e-a830-4069-a3fd-4c43c149b943\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch" Oct 30 18:22:49.828618 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:49.828572 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfn87\" (UniqueName: \"kubernetes.io/projected/00eedd3e-a830-4069-a3fd-4c43c149b943-kube-api-access-cfn87\") pod \"maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch\" (UID: \"00eedd3e-a830-4069-a3fd-4c43c149b943\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch" Oct 30 18:22:49.828773 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:49.828757 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00eedd3e-a830-4069-a3fd-4c43c149b943-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch\" (UID: \"00eedd3e-a830-4069-a3fd-4c43c149b943\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch" Oct 30 18:22:49.828830 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:49.828780 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00eedd3e-a830-4069-a3fd-4c43c149b943-metrics-certs\") pod \"maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch\" (UID: \"00eedd3e-a830-4069-a3fd-4c43c149b943\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch" Oct 30 18:22:49.930243 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:49.930211 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfn87\" (UniqueName: \"kubernetes.io/projected/00eedd3e-a830-4069-a3fd-4c43c149b943-kube-api-access-cfn87\") pod \"maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch\" (UID: \"00eedd3e-a830-4069-a3fd-4c43c149b943\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch" Oct 30 18:22:49.930455 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:49.930383 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00eedd3e-a830-4069-a3fd-4c43c149b943-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch\" (UID: \"00eedd3e-a830-4069-a3fd-4c43c149b943\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch" Oct 30 18:22:49.930455 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:49.930402 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00eedd3e-a830-4069-a3fd-4c43c149b943-metrics-certs\") pod \"maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch\" (UID: \"00eedd3e-a830-4069-a3fd-4c43c149b943\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch" Oct 30 18:22:49.930581 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:49.930524 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00eedd3e-a830-4069-a3fd-4c43c149b943-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch\" (UID: \"00eedd3e-a830-4069-a3fd-4c43c149b943\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch" Oct 30 18:22:49.931216 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:49.931187 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00eedd3e-a830-4069-a3fd-4c43c149b943-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch\" (UID: \"00eedd3e-a830-4069-a3fd-4c43c149b943\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch" Oct 30 18:22:49.933229 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:49.933197 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00eedd3e-a830-4069-a3fd-4c43c149b943-metrics-certs\") pod \"maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch\" (UID: \"00eedd3e-a830-4069-a3fd-4c43c149b943\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch" Oct 30 18:22:49.933373 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:49.933197 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00eedd3e-a830-4069-a3fd-4c43c149b943-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch\" (UID: \"00eedd3e-a830-4069-a3fd-4c43c149b943\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch" Oct 30 18:22:49.981545 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:49.958079 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfn87\" (UniqueName: \"kubernetes.io/projected/00eedd3e-a830-4069-a3fd-4c43c149b943-kube-api-access-cfn87\") pod \"maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch\" (UID: \"00eedd3e-a830-4069-a3fd-4c43c149b943\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch" Oct 30 18:22:50.038404 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:50.037844 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch" Oct 30 18:22:50.360472 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:22:50.360442 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00eedd3e_a830_4069_a3fd_4c43c149b943.slice/crio-526474a3d9cf60ec820bdea1f6765cb4f342650cc157e8acc06dc66e9aa68ebc WatchSource:0}: Error finding container 526474a3d9cf60ec820bdea1f6765cb4f342650cc157e8acc06dc66e9aa68ebc: Status 404 returned error can't find the container with id 526474a3d9cf60ec820bdea1f6765cb4f342650cc157e8acc06dc66e9aa68ebc Oct 30 18:22:50.362432 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:50.362402 4455 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:22:50.362888 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:50.362862 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch"] Oct 30 18:22:51.010606 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:51.010565 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch" event={"ID":"00eedd3e-a830-4069-a3fd-4c43c149b943","Type":"ContainerStarted","Data":"526474a3d9cf60ec820bdea1f6765cb4f342650cc157e8acc06dc66e9aa68ebc"} Oct 30 18:22:52.032745 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:52.032707 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch" event={"ID":"00eedd3e-a830-4069-a3fd-4c43c149b943","Type":"ContainerStarted","Data":"59babe6be3c2d3a6c62bc74301f084b559b1746d5a0bffd99388015e47a3bb3b"} Oct 30 18:22:52.033821 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:52.033796 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch" Oct 30 18:22:52.084806 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:52.084735 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch" podStartSLOduration=2.606764497 podStartE2EDuration="3.084716469s" totalImagesPullingTime="477.951972ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:52.080580148 +0000 UTC m=+1824.568217183" watchObservedRunningTime="2026-10-30 18:22:52.084716469 +0000 UTC m=+1824.572353511" Oct 30 18:22:53.227687 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:53.227652 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp"] Oct 30 18:22:53.228120 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:53.227898 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp" podUID="dd45faeb-139e-4756-9b91-4ed08d919e39" containerName="maas-api" containerID="cri-o://d16af3f8d3d132f6148ee5f98a780ef9d37055dba47040a6a030eacfb1338b2c" gracePeriod=30 Oct 30 18:22:53.547052 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:53.547023 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp" Oct 30 18:22:53.679898 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:53.679863 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd45faeb-139e-4756-9b91-4ed08d919e39-kube-api-access-js8zc\" (UniqueName: \"kubernetes.io/projected/dd45faeb-139e-4756-9b91-4ed08d919e39-kube-api-access-js8zc\") pod \"dd45faeb-139e-4756-9b91-4ed08d919e39\" (UID: \"dd45faeb-139e-4756-9b91-4ed08d919e39\") " Oct 30 18:22:53.680109 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:53.680011 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd45faeb-139e-4756-9b91-4ed08d919e39-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd45faeb-139e-4756-9b91-4ed08d919e39-openshift-service-ca\") pod \"dd45faeb-139e-4756-9b91-4ed08d919e39\" (UID: \"dd45faeb-139e-4756-9b91-4ed08d919e39\") " Oct 30 18:22:53.680109 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:53.680093 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd45faeb-139e-4756-9b91-4ed08d919e39-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd45faeb-139e-4756-9b91-4ed08d919e39-maas-api-tls\") pod \"dd45faeb-139e-4756-9b91-4ed08d919e39\" (UID: \"dd45faeb-139e-4756-9b91-4ed08d919e39\") " Oct 30 18:22:53.680273 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:53.680259 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd45faeb-139e-4756-9b91-4ed08d919e39-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd45faeb-139e-4756-9b91-4ed08d919e39-metrics-certs\") pod \"dd45faeb-139e-4756-9b91-4ed08d919e39\" (UID: \"dd45faeb-139e-4756-9b91-4ed08d919e39\") " Oct 30 18:22:53.680440 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:53.680409 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd45faeb-139e-4756-9b91-4ed08d919e39-openshift-service-ca" pod "dd45faeb-139e-4756-9b91-4ed08d919e39" (UID: "dd45faeb-139e-4756-9b91-4ed08d919e39"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:53.680954 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:53.680930 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd45faeb-139e-4756-9b91-4ed08d919e39-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:22:53.683084 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:53.683057 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd45faeb-139e-4756-9b91-4ed08d919e39-maas-api-tls" pod "dd45faeb-139e-4756-9b91-4ed08d919e39" (UID: "dd45faeb-139e-4756-9b91-4ed08d919e39"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:22:53.683195 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:53.683117 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd45faeb-139e-4756-9b91-4ed08d919e39-metrics-certs" pod "dd45faeb-139e-4756-9b91-4ed08d919e39" (UID: "dd45faeb-139e-4756-9b91-4ed08d919e39"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:22:53.683195 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:53.683173 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd45faeb-139e-4756-9b91-4ed08d919e39-kube-api-access-js8zc" pod "dd45faeb-139e-4756-9b91-4ed08d919e39" (UID: "dd45faeb-139e-4756-9b91-4ed08d919e39"). InnerVolumeSpecName "kube-api-access-js8zc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:53.848412 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:53.838659 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js8zc\" (UniqueName: \"kubernetes.io/projected/dd45faeb-139e-4756-9b91-4ed08d919e39-kube-api-access-js8zc\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:22:53.848412 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:53.838696 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd45faeb-139e-4756-9b91-4ed08d919e39-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:22:53.848412 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:53.838713 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd45faeb-139e-4756-9b91-4ed08d919e39-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:22:54.049781 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:54.049746 4455 generic.go:356] "Generic (PLEG): container finished" podID="dd45faeb-139e-4756-9b91-4ed08d919e39" containerID="d16af3f8d3d132f6148ee5f98a780ef9d37055dba47040a6a030eacfb1338b2c" exitCode=0 Oct 30 18:22:54.049976 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:54.049807 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp" Oct 30 18:22:54.049976 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:54.049831 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp" event={"ID":"dd45faeb-139e-4756-9b91-4ed08d919e39","Type":"ContainerDied","Data":"d16af3f8d3d132f6148ee5f98a780ef9d37055dba47040a6a030eacfb1338b2c"} Oct 30 18:22:54.049976 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:54.049888 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp" event={"ID":"dd45faeb-139e-4756-9b91-4ed08d919e39","Type":"ContainerDied","Data":"4052feb8dbe0fcb71f831ab2c3a284e04b08a1c4b0a6f58c329275568bb1593e"} Oct 30 18:22:54.049976 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:54.049932 4455 scope.go:122] "RemoveContainer" containerID="d16af3f8d3d132f6148ee5f98a780ef9d37055dba47040a6a030eacfb1338b2c" Oct 30 18:22:54.062153 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:54.062133 4455 scope.go:122] "RemoveContainer" containerID="d16af3f8d3d132f6148ee5f98a780ef9d37055dba47040a6a030eacfb1338b2c" Oct 30 18:22:54.062425 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:22:54.062402 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d16af3f8d3d132f6148ee5f98a780ef9d37055dba47040a6a030eacfb1338b2c\": container with ID starting with d16af3f8d3d132f6148ee5f98a780ef9d37055dba47040a6a030eacfb1338b2c not found: ID does not exist" containerID="d16af3f8d3d132f6148ee5f98a780ef9d37055dba47040a6a030eacfb1338b2c" Oct 30 18:22:54.062491 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:54.062434 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d16af3f8d3d132f6148ee5f98a780ef9d37055dba47040a6a030eacfb1338b2c"} err="failed to get container status \"d16af3f8d3d132f6148ee5f98a780ef9d37055dba47040a6a030eacfb1338b2c\": rpc error: code = NotFound desc = could not find container \"d16af3f8d3d132f6148ee5f98a780ef9d37055dba47040a6a030eacfb1338b2c\": container with ID starting with d16af3f8d3d132f6148ee5f98a780ef9d37055dba47040a6a030eacfb1338b2c not found: ID does not exist" Oct 30 18:22:54.113411 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:54.113280 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp"] Oct 30 18:22:54.118795 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:54.118765 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d951c4-cbd897b9c-vqxmp"] Oct 30 18:22:55.636109 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:55.636074 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd45faeb-139e-4756-9b91-4ed08d919e39" path="/var/lib/kubelet/pods/dd45faeb-139e-4756-9b91-4ed08d919e39/volumes" Oct 30 18:22:55.932718 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:55.932644 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca5f33-54dcbfd498-fvclr" Oct 30 18:22:55.932718 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:55.932715 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca5f33-546df68d7-nnnlz" Oct 30 18:22:57.987345 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:57.983550 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ef208a-9ccb8d8f5-jqhpw" Oct 30 18:22:57.991966 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:57.991937 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-ef208a-6c5cd4b4d-wrtmz" Oct 30 18:22:59.051838 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:22:59.051795 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch" podUID="00eedd3e-a830-4069-a3fd-4c43c149b943" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.157:8443/health\": dial tcp 10.134.0.157:8443: connect: connection refused" Oct 30 18:23:00.405790 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:23:00.405748 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:23:01.356890 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:01.356851 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78"] Oct 30 18:23:01.357685 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:01.357669 4455 cpu_manager.go:414] "containerMap: removing container" podUID="dd45faeb-139e-4756-9b91-4ed08d919e39" containerName="maas-api" Oct 30 18:23:01.357755 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:01.357687 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd45faeb-139e-4756-9b91-4ed08d919e39" containerName="maas-api" Oct 30 18:23:01.357800 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:01.357792 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd45faeb-139e-4756-9b91-4ed08d919e39" containerName="maas-api" Oct 30 18:23:01.361735 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:01.361680 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78" Oct 30 18:23:01.388296 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:01.388271 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-ef208a-serving-cert\"" Oct 30 18:23:01.392651 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:01.392626 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78"] Oct 30 18:23:01.506022 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:01.505986 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1080e9c-6b0d-4aef-90a6-e041970b9a35-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78\" (UID: \"a1080e9c-6b0d-4aef-90a6-e041970b9a35\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78" Oct 30 18:23:01.506522 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:01.506034 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1080e9c-6b0d-4aef-90a6-e041970b9a35-metrics-certs\") pod \"maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78\" (UID: \"a1080e9c-6b0d-4aef-90a6-e041970b9a35\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78" Oct 30 18:23:01.506522 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:01.506229 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkhf7\" (UniqueName: \"kubernetes.io/projected/a1080e9c-6b0d-4aef-90a6-e041970b9a35-kube-api-access-kkhf7\") pod \"maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78\" (UID: \"a1080e9c-6b0d-4aef-90a6-e041970b9a35\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78" Oct 30 18:23:01.506522 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:01.506289 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1080e9c-6b0d-4aef-90a6-e041970b9a35-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78\" (UID: \"a1080e9c-6b0d-4aef-90a6-e041970b9a35\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78" Oct 30 18:23:01.623394 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:01.616310 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1080e9c-6b0d-4aef-90a6-e041970b9a35-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78\" (UID: \"a1080e9c-6b0d-4aef-90a6-e041970b9a35\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78" Oct 30 18:23:01.623394 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:01.616406 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1080e9c-6b0d-4aef-90a6-e041970b9a35-metrics-certs\") pod \"maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78\" (UID: \"a1080e9c-6b0d-4aef-90a6-e041970b9a35\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78" Oct 30 18:23:01.623394 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:01.616542 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkhf7\" (UniqueName: \"kubernetes.io/projected/a1080e9c-6b0d-4aef-90a6-e041970b9a35-kube-api-access-kkhf7\") pod \"maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78\" (UID: \"a1080e9c-6b0d-4aef-90a6-e041970b9a35\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78" Oct 30 18:23:01.623394 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:01.616593 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1080e9c-6b0d-4aef-90a6-e041970b9a35-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78\" (UID: \"a1080e9c-6b0d-4aef-90a6-e041970b9a35\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78" Oct 30 18:23:01.623394 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:01.617568 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1080e9c-6b0d-4aef-90a6-e041970b9a35-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78\" (UID: \"a1080e9c-6b0d-4aef-90a6-e041970b9a35\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78" Oct 30 18:23:01.670628 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:01.648691 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1080e9c-6b0d-4aef-90a6-e041970b9a35-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78\" (UID: \"a1080e9c-6b0d-4aef-90a6-e041970b9a35\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78" Oct 30 18:23:01.670628 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:01.658044 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1080e9c-6b0d-4aef-90a6-e041970b9a35-metrics-certs\") pod \"maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78\" (UID: \"a1080e9c-6b0d-4aef-90a6-e041970b9a35\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78" Oct 30 18:23:01.753632 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:01.743058 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkhf7\" (UniqueName: \"kubernetes.io/projected/a1080e9c-6b0d-4aef-90a6-e041970b9a35-kube-api-access-kkhf7\") pod \"maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78\" (UID: \"a1080e9c-6b0d-4aef-90a6-e041970b9a35\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78" Oct 30 18:23:01.976920 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:01.976879 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78" Oct 30 18:23:02.128873 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:02.128835 4455 generic.go:356] "Generic (PLEG): container finished" podID="00eedd3e-a830-4069-a3fd-4c43c149b943" containerID="59babe6be3c2d3a6c62bc74301f084b559b1746d5a0bffd99388015e47a3bb3b" exitCode=1 Oct 30 18:23:02.129060 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:02.128914 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch" event={"ID":"00eedd3e-a830-4069-a3fd-4c43c149b943","Type":"ContainerDied","Data":"59babe6be3c2d3a6c62bc74301f084b559b1746d5a0bffd99388015e47a3bb3b"} Oct 30 18:23:02.129458 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:02.129436 4455 scope.go:122] "RemoveContainer" containerID="59babe6be3c2d3a6c62bc74301f084b559b1746d5a0bffd99388015e47a3bb3b" Oct 30 18:23:02.284728 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:23:02.284702 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1080e9c_6b0d_4aef_90a6_e041970b9a35.slice/crio-00ec3f1292fb64224f9fd893ed0450b1de7214d4091ed4d9eb7b6dc6f2ccf37c WatchSource:0}: Error finding container 00ec3f1292fb64224f9fd893ed0450b1de7214d4091ed4d9eb7b6dc6f2ccf37c: Status 404 returned error can't find the container with id 00ec3f1292fb64224f9fd893ed0450b1de7214d4091ed4d9eb7b6dc6f2ccf37c Oct 30 18:23:02.286906 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:02.286885 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78"] Oct 30 18:23:03.141790 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:03.141746 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch" event={"ID":"00eedd3e-a830-4069-a3fd-4c43c149b943","Type":"ContainerStarted","Data":"9e6597864e5b0562f710b2455bf03a58cca8760c32508aee8482c4b8fc46192f"} Oct 30 18:23:03.142248 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:03.142003 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch" Oct 30 18:23:03.143575 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:03.143549 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78" event={"ID":"a1080e9c-6b0d-4aef-90a6-e041970b9a35","Type":"ContainerStarted","Data":"f61d29e9ecb27f3531335c5aae3f8ad9ef5aeb5f411c8f091d4d81b7db3a6047"} Oct 30 18:23:03.143747 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:03.143588 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78" event={"ID":"a1080e9c-6b0d-4aef-90a6-e041970b9a35","Type":"ContainerStarted","Data":"00ec3f1292fb64224f9fd893ed0450b1de7214d4091ed4d9eb7b6dc6f2ccf37c"} Oct 30 18:23:03.143747 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:03.143622 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78" Oct 30 18:23:09.160449 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:09.160365 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch" Oct 30 18:23:09.161417 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:09.161396 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78" Oct 30 18:23:09.207663 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:09.207611 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78" podStartSLOduration=7.893484596 podStartE2EDuration="8.207597268s" totalImagesPullingTime="314.112672ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:03.261709635 +0000 UTC m=+1835.749346682" watchObservedRunningTime="2026-10-30 18:23:09.207597268 +0000 UTC m=+1841.695234354" Oct 30 18:23:24.297313 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:24.297264 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz"] Oct 30 18:23:24.298005 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:24.297511 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz" podUID="466c7c11-2221-4749-bba1-d0f2383c66d2" containerName="maas-api" containerID="cri-o://65d681be6e180e4267af0626e72a02dff490c0a61f7b59322c3db7cc52bfb00d" gracePeriod=30 Oct 30 18:23:24.467106 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:24.467065 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-394a31-7d9db95ddb-8xl6n"] Oct 30 18:23:24.467346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:24.467304 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-7d9db95ddb-8xl6n" podUID="97e9aed5-3f20-4301-b832-a78b35c9fb71" containerName="payload-processing" containerID="cri-o://daa58f58910292e97b52bb62ac06f9c31d2e5d54231a4be97b68270fb0e8b9ef" gracePeriod=30 Oct 30 18:23:24.486783 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:24.486756 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-758c6cf78c-glwf2"] Oct 30 18:23:24.487142 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:24.487109 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-758c6cf78c-glwf2" podUID="d9e693c8-eff4-4afd-8c2a-3009b6c0c27f" containerName="payload-pre-processing" containerID="cri-o://140b5d1fcb6bbbb0e9677d7f92db6f17a0dd2df891f5ff981904cb3ae1e352c2" gracePeriod=30 Oct 30 18:23:24.657803 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:24.657427 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz" Oct 30 18:23:24.774238 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:24.774196 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/466c7c11-2221-4749-bba1-d0f2383c66d2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/466c7c11-2221-4749-bba1-d0f2383c66d2-metrics-certs\") pod \"466c7c11-2221-4749-bba1-d0f2383c66d2\" (UID: \"466c7c11-2221-4749-bba1-d0f2383c66d2\") " Oct 30 18:23:24.774429 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:24.774245 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/466c7c11-2221-4749-bba1-d0f2383c66d2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/466c7c11-2221-4749-bba1-d0f2383c66d2-maas-api-tls\") pod \"466c7c11-2221-4749-bba1-d0f2383c66d2\" (UID: \"466c7c11-2221-4749-bba1-d0f2383c66d2\") " Oct 30 18:23:24.774429 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:24.774308 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/466c7c11-2221-4749-bba1-d0f2383c66d2-kube-api-access-fjl8n\" (UniqueName: \"kubernetes.io/projected/466c7c11-2221-4749-bba1-d0f2383c66d2-kube-api-access-fjl8n\") pod \"466c7c11-2221-4749-bba1-d0f2383c66d2\" (UID: \"466c7c11-2221-4749-bba1-d0f2383c66d2\") " Oct 30 18:23:24.774648 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:24.774622 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/466c7c11-2221-4749-bba1-d0f2383c66d2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/466c7c11-2221-4749-bba1-d0f2383c66d2-openshift-service-ca\") pod \"466c7c11-2221-4749-bba1-d0f2383c66d2\" (UID: \"466c7c11-2221-4749-bba1-d0f2383c66d2\") " Oct 30 18:23:24.775511 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:24.775068 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/466c7c11-2221-4749-bba1-d0f2383c66d2-openshift-service-ca" pod "466c7c11-2221-4749-bba1-d0f2383c66d2" (UID: "466c7c11-2221-4749-bba1-d0f2383c66d2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:23:24.775511 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:24.775404 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/466c7c11-2221-4749-bba1-d0f2383c66d2-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:23:24.777901 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:24.777871 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/466c7c11-2221-4749-bba1-d0f2383c66d2-metrics-certs" pod "466c7c11-2221-4749-bba1-d0f2383c66d2" (UID: "466c7c11-2221-4749-bba1-d0f2383c66d2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:23:24.777980 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:24.777909 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/466c7c11-2221-4749-bba1-d0f2383c66d2-maas-api-tls" pod "466c7c11-2221-4749-bba1-d0f2383c66d2" (UID: "466c7c11-2221-4749-bba1-d0f2383c66d2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:23:24.778948 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:24.778927 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/466c7c11-2221-4749-bba1-d0f2383c66d2-kube-api-access-fjl8n" pod "466c7c11-2221-4749-bba1-d0f2383c66d2" (UID: "466c7c11-2221-4749-bba1-d0f2383c66d2"). InnerVolumeSpecName "kube-api-access-fjl8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:24.890347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:24.879500 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/466c7c11-2221-4749-bba1-d0f2383c66d2-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:23:24.890347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:24.879538 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/466c7c11-2221-4749-bba1-d0f2383c66d2-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:23:24.890347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:24.879552 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjl8n\" (UniqueName: \"kubernetes.io/projected/466c7c11-2221-4749-bba1-d0f2383c66d2-kube-api-access-fjl8n\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:23:25.002360 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.001513 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-758c6cf78c-glwf2" Oct 30 18:23:25.112397 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.111846 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9e693c8-eff4-4afd-8c2a-3009b6c0c27f-kube-api-access-zwpch\" (UniqueName: \"kubernetes.io/projected/d9e693c8-eff4-4afd-8c2a-3009b6c0c27f-kube-api-access-zwpch\") pod \"d9e693c8-eff4-4afd-8c2a-3009b6c0c27f\" (UID: \"d9e693c8-eff4-4afd-8c2a-3009b6c0c27f\") " Oct 30 18:23:25.112397 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.112040 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9e693c8-eff4-4afd-8c2a-3009b6c0c27f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e693c8-eff4-4afd-8c2a-3009b6c0c27f-plugins-config-volume\") pod \"d9e693c8-eff4-4afd-8c2a-3009b6c0c27f\" (UID: \"d9e693c8-eff4-4afd-8c2a-3009b6c0c27f\") " Oct 30 18:23:25.121071 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.114836 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9e693c8-eff4-4afd-8c2a-3009b6c0c27f-plugins-config-volume" pod "d9e693c8-eff4-4afd-8c2a-3009b6c0c27f" (UID: "d9e693c8-eff4-4afd-8c2a-3009b6c0c27f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:23:25.122853 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.122817 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e693c8-eff4-4afd-8c2a-3009b6c0c27f-kube-api-access-zwpch" pod "d9e693c8-eff4-4afd-8c2a-3009b6c0c27f" (UID: "d9e693c8-eff4-4afd-8c2a-3009b6c0c27f"). InnerVolumeSpecName "kube-api-access-zwpch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:25.154222 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.154185 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-7d9db95ddb-8xl6n" Oct 30 18:23:25.226346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.224598 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwpch\" (UniqueName: \"kubernetes.io/projected/d9e693c8-eff4-4afd-8c2a-3009b6c0c27f-kube-api-access-zwpch\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:23:25.226346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.224636 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e693c8-eff4-4afd-8c2a-3009b6c0c27f-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:23:25.325237 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.325206 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97e9aed5-3f20-4301-b832-a78b35c9fb71-kube-api-access-hj9nb\" (UniqueName: \"kubernetes.io/projected/97e9aed5-3f20-4301-b832-a78b35c9fb71-kube-api-access-hj9nb\") pod \"97e9aed5-3f20-4301-b832-a78b35c9fb71\" (UID: \"97e9aed5-3f20-4301-b832-a78b35c9fb71\") " Oct 30 18:23:25.325765 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.325261 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97e9aed5-3f20-4301-b832-a78b35c9fb71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97e9aed5-3f20-4301-b832-a78b35c9fb71-plugins-config-volume\") pod \"97e9aed5-3f20-4301-b832-a78b35c9fb71\" (UID: \"97e9aed5-3f20-4301-b832-a78b35c9fb71\") " Oct 30 18:23:25.325765 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.325736 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97e9aed5-3f20-4301-b832-a78b35c9fb71-plugins-config-volume" pod "97e9aed5-3f20-4301-b832-a78b35c9fb71" (UID: "97e9aed5-3f20-4301-b832-a78b35c9fb71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:23:25.325921 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.325897 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97e9aed5-3f20-4301-b832-a78b35c9fb71-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:23:25.328513 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.328471 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97e9aed5-3f20-4301-b832-a78b35c9fb71-kube-api-access-hj9nb" pod "97e9aed5-3f20-4301-b832-a78b35c9fb71" (UID: "97e9aed5-3f20-4301-b832-a78b35c9fb71"). InnerVolumeSpecName "kube-api-access-hj9nb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:25.408217 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.408179 4455 generic.go:356] "Generic (PLEG): container finished" podID="466c7c11-2221-4749-bba1-d0f2383c66d2" containerID="65d681be6e180e4267af0626e72a02dff490c0a61f7b59322c3db7cc52bfb00d" exitCode=0 Oct 30 18:23:25.408435 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.408248 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz" Oct 30 18:23:25.408435 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.408260 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz" event={"ID":"466c7c11-2221-4749-bba1-d0f2383c66d2","Type":"ContainerDied","Data":"65d681be6e180e4267af0626e72a02dff490c0a61f7b59322c3db7cc52bfb00d"} Oct 30 18:23:25.408435 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.408377 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz" event={"ID":"466c7c11-2221-4749-bba1-d0f2383c66d2","Type":"ContainerDied","Data":"cb51243ddb5b149a36e5979b772b7bdc4b9e32bf958c9cabed0333cc3dbff3a6"} Oct 30 18:23:25.408435 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.408412 4455 scope.go:122] "RemoveContainer" containerID="65d681be6e180e4267af0626e72a02dff490c0a61f7b59322c3db7cc52bfb00d" Oct 30 18:23:25.410418 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.410391 4455 generic.go:356] "Generic (PLEG): container finished" podID="97e9aed5-3f20-4301-b832-a78b35c9fb71" containerID="daa58f58910292e97b52bb62ac06f9c31d2e5d54231a4be97b68270fb0e8b9ef" exitCode=0 Oct 30 18:23:25.410553 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.410433 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-7d9db95ddb-8xl6n" Oct 30 18:23:25.410553 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.410473 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-7d9db95ddb-8xl6n" event={"ID":"97e9aed5-3f20-4301-b832-a78b35c9fb71","Type":"ContainerDied","Data":"daa58f58910292e97b52bb62ac06f9c31d2e5d54231a4be97b68270fb0e8b9ef"} Oct 30 18:23:25.410553 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.410532 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-394a31-7d9db95ddb-8xl6n" event={"ID":"97e9aed5-3f20-4301-b832-a78b35c9fb71","Type":"ContainerDied","Data":"a49f6d06ee16ca3532b9ca6bd25af44bc7a74ea7bda2928e838ee406fb4c55a2"} Oct 30 18:23:25.412438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.412401 4455 generic.go:356] "Generic (PLEG): container finished" podID="d9e693c8-eff4-4afd-8c2a-3009b6c0c27f" containerID="140b5d1fcb6bbbb0e9677d7f92db6f17a0dd2df891f5ff981904cb3ae1e352c2" exitCode=0 Oct 30 18:23:25.412544 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.412510 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-758c6cf78c-glwf2" event={"ID":"d9e693c8-eff4-4afd-8c2a-3009b6c0c27f","Type":"ContainerDied","Data":"140b5d1fcb6bbbb0e9677d7f92db6f17a0dd2df891f5ff981904cb3ae1e352c2"} Oct 30 18:23:25.412619 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.412548 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-758c6cf78c-glwf2" event={"ID":"d9e693c8-eff4-4afd-8c2a-3009b6c0c27f","Type":"ContainerDied","Data":"c6f0bc0b701fcad0f7d42a6cf22ec32fdd1c02afe66d7fba7299aadc7925151d"} Oct 30 18:23:25.412674 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.412648 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-758c6cf78c-glwf2" Oct 30 18:23:25.424115 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.424046 4455 scope.go:122] "RemoveContainer" containerID="65d681be6e180e4267af0626e72a02dff490c0a61f7b59322c3db7cc52bfb00d" Oct 30 18:23:25.425218 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:23:25.425195 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65d681be6e180e4267af0626e72a02dff490c0a61f7b59322c3db7cc52bfb00d\": container with ID starting with 65d681be6e180e4267af0626e72a02dff490c0a61f7b59322c3db7cc52bfb00d not found: ID does not exist" containerID="65d681be6e180e4267af0626e72a02dff490c0a61f7b59322c3db7cc52bfb00d" Oct 30 18:23:25.425362 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.425224 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65d681be6e180e4267af0626e72a02dff490c0a61f7b59322c3db7cc52bfb00d"} err="failed to get container status \"65d681be6e180e4267af0626e72a02dff490c0a61f7b59322c3db7cc52bfb00d\": rpc error: code = NotFound desc = could not find container \"65d681be6e180e4267af0626e72a02dff490c0a61f7b59322c3db7cc52bfb00d\": container with ID starting with 65d681be6e180e4267af0626e72a02dff490c0a61f7b59322c3db7cc52bfb00d not found: ID does not exist" Oct 30 18:23:25.425362 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.425239 4455 scope.go:122] "RemoveContainer" containerID="daa58f58910292e97b52bb62ac06f9c31d2e5d54231a4be97b68270fb0e8b9ef" Oct 30 18:23:25.427261 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.427238 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj9nb\" (UniqueName: \"kubernetes.io/projected/97e9aed5-3f20-4301-b832-a78b35c9fb71-kube-api-access-hj9nb\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:23:25.436144 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.436127 4455 scope.go:122] "RemoveContainer" containerID="ad11f172ba6b49e3b39780038ab7e5c2b1f1401488cd72dc88b57b4ae4f01464" Oct 30 18:23:25.450874 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.450851 4455 scope.go:122] "RemoveContainer" containerID="daa58f58910292e97b52bb62ac06f9c31d2e5d54231a4be97b68270fb0e8b9ef" Oct 30 18:23:25.451380 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:23:25.451307 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daa58f58910292e97b52bb62ac06f9c31d2e5d54231a4be97b68270fb0e8b9ef\": container with ID starting with daa58f58910292e97b52bb62ac06f9c31d2e5d54231a4be97b68270fb0e8b9ef not found: ID does not exist" containerID="daa58f58910292e97b52bb62ac06f9c31d2e5d54231a4be97b68270fb0e8b9ef" Oct 30 18:23:25.451477 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.451392 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daa58f58910292e97b52bb62ac06f9c31d2e5d54231a4be97b68270fb0e8b9ef"} err="failed to get container status \"daa58f58910292e97b52bb62ac06f9c31d2e5d54231a4be97b68270fb0e8b9ef\": rpc error: code = NotFound desc = could not find container \"daa58f58910292e97b52bb62ac06f9c31d2e5d54231a4be97b68270fb0e8b9ef\": container with ID starting with daa58f58910292e97b52bb62ac06f9c31d2e5d54231a4be97b68270fb0e8b9ef not found: ID does not exist" Oct 30 18:23:25.451477 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.451416 4455 scope.go:122] "RemoveContainer" containerID="ad11f172ba6b49e3b39780038ab7e5c2b1f1401488cd72dc88b57b4ae4f01464" Oct 30 18:23:25.451738 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:23:25.451717 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad11f172ba6b49e3b39780038ab7e5c2b1f1401488cd72dc88b57b4ae4f01464\": container with ID starting with ad11f172ba6b49e3b39780038ab7e5c2b1f1401488cd72dc88b57b4ae4f01464 not found: ID does not exist" containerID="ad11f172ba6b49e3b39780038ab7e5c2b1f1401488cd72dc88b57b4ae4f01464" Oct 30 18:23:25.451800 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.451745 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad11f172ba6b49e3b39780038ab7e5c2b1f1401488cd72dc88b57b4ae4f01464"} err="failed to get container status \"ad11f172ba6b49e3b39780038ab7e5c2b1f1401488cd72dc88b57b4ae4f01464\": rpc error: code = NotFound desc = could not find container \"ad11f172ba6b49e3b39780038ab7e5c2b1f1401488cd72dc88b57b4ae4f01464\": container with ID starting with ad11f172ba6b49e3b39780038ab7e5c2b1f1401488cd72dc88b57b4ae4f01464 not found: ID does not exist" Oct 30 18:23:25.451800 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.451760 4455 scope.go:122] "RemoveContainer" containerID="140b5d1fcb6bbbb0e9677d7f92db6f17a0dd2df891f5ff981904cb3ae1e352c2" Oct 30 18:23:25.462888 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.462869 4455 scope.go:122] "RemoveContainer" containerID="140b5d1fcb6bbbb0e9677d7f92db6f17a0dd2df891f5ff981904cb3ae1e352c2" Oct 30 18:23:25.463159 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:23:25.463140 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"140b5d1fcb6bbbb0e9677d7f92db6f17a0dd2df891f5ff981904cb3ae1e352c2\": container with ID starting with 140b5d1fcb6bbbb0e9677d7f92db6f17a0dd2df891f5ff981904cb3ae1e352c2 not found: ID does not exist" containerID="140b5d1fcb6bbbb0e9677d7f92db6f17a0dd2df891f5ff981904cb3ae1e352c2" Oct 30 18:23:25.463220 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.463166 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"140b5d1fcb6bbbb0e9677d7f92db6f17a0dd2df891f5ff981904cb3ae1e352c2"} err="failed to get container status \"140b5d1fcb6bbbb0e9677d7f92db6f17a0dd2df891f5ff981904cb3ae1e352c2\": rpc error: code = NotFound desc = could not find container \"140b5d1fcb6bbbb0e9677d7f92db6f17a0dd2df891f5ff981904cb3ae1e352c2\": container with ID starting with 140b5d1fcb6bbbb0e9677d7f92db6f17a0dd2df891f5ff981904cb3ae1e352c2 not found: ID does not exist" Oct 30 18:23:25.481335 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.481298 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz"] Oct 30 18:23:25.487277 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.487254 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-394a31-7c6c4847-hswjz"] Oct 30 18:23:25.556130 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.556096 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-758c6cf78c-glwf2"] Oct 30 18:23:25.574409 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.574373 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-394a31-758c6cf78c-glwf2"] Oct 30 18:23:25.617614 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.617578 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-394a31-7d9db95ddb-8xl6n"] Oct 30 18:23:25.621192 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.621168 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-394a31-7d9db95ddb-8xl6n"] Oct 30 18:23:25.637095 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.637031 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="466c7c11-2221-4749-bba1-d0f2383c66d2" path="/var/lib/kubelet/pods/466c7c11-2221-4749-bba1-d0f2383c66d2/volumes" Oct 30 18:23:25.637495 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.637480 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97e9aed5-3f20-4301-b832-a78b35c9fb71" path="/var/lib/kubelet/pods/97e9aed5-3f20-4301-b832-a78b35c9fb71/volumes" Oct 30 18:23:25.637873 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:25.637861 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9e693c8-eff4-4afd-8c2a-3009b6c0c27f" path="/var/lib/kubelet/pods/d9e693c8-eff4-4afd-8c2a-3009b6c0c27f/volumes" Oct 30 18:23:30.016250 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:30.015563 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr"] Oct 30 18:23:30.016250 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:30.015860 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr" podUID="a99b4f25-e266-43dd-bd25-576847ed1b4a" containerName="maas-api" containerID="cri-o://415686924fb73625eeb0794e47272a3f9124bc0d7a53acca13da3c081828c726" gracePeriod=30 Oct 30 18:23:30.585081 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:23:30.585037 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:23:30.604281 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:30.603824 4455 generic.go:356] "Generic (PLEG): container finished" podID="a99b4f25-e266-43dd-bd25-576847ed1b4a" containerID="415686924fb73625eeb0794e47272a3f9124bc0d7a53acca13da3c081828c726" exitCode=0 Oct 30 18:23:30.604281 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:30.604081 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr" event={"ID":"a99b4f25-e266-43dd-bd25-576847ed1b4a","Type":"ContainerDied","Data":"415686924fb73625eeb0794e47272a3f9124bc0d7a53acca13da3c081828c726"} Oct 30 18:23:30.662270 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:30.652717 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr" Oct 30 18:23:30.783463 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:30.762629 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a99b4f25-e266-43dd-bd25-576847ed1b4a-kube-api-access-494sp\" (UniqueName: \"kubernetes.io/projected/a99b4f25-e266-43dd-bd25-576847ed1b4a-kube-api-access-494sp\") pod \"a99b4f25-e266-43dd-bd25-576847ed1b4a\" (UID: \"a99b4f25-e266-43dd-bd25-576847ed1b4a\") " Oct 30 18:23:30.783959 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:30.783922 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a99b4f25-e266-43dd-bd25-576847ed1b4a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a99b4f25-e266-43dd-bd25-576847ed1b4a-openshift-service-ca\") pod \"a99b4f25-e266-43dd-bd25-576847ed1b4a\" (UID: \"a99b4f25-e266-43dd-bd25-576847ed1b4a\") " Oct 30 18:23:30.784682 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:30.784657 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a99b4f25-e266-43dd-bd25-576847ed1b4a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a99b4f25-e266-43dd-bd25-576847ed1b4a-maas-api-tls\") pod \"a99b4f25-e266-43dd-bd25-576847ed1b4a\" (UID: \"a99b4f25-e266-43dd-bd25-576847ed1b4a\") " Oct 30 18:23:30.815557 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:30.814494 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a99b4f25-e266-43dd-bd25-576847ed1b4a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a99b4f25-e266-43dd-bd25-576847ed1b4a-metrics-certs\") pod \"a99b4f25-e266-43dd-bd25-576847ed1b4a\" (UID: \"a99b4f25-e266-43dd-bd25-576847ed1b4a\") " Oct 30 18:23:30.818743 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:30.784507 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a99b4f25-e266-43dd-bd25-576847ed1b4a-openshift-service-ca" pod "a99b4f25-e266-43dd-bd25-576847ed1b4a" (UID: "a99b4f25-e266-43dd-bd25-576847ed1b4a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:23:30.853303 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:30.853227 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a99b4f25-e266-43dd-bd25-576847ed1b4a-maas-api-tls" pod "a99b4f25-e266-43dd-bd25-576847ed1b4a" (UID: "a99b4f25-e266-43dd-bd25-576847ed1b4a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:23:30.879830 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:30.879782 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a99b4f25-e266-43dd-bd25-576847ed1b4a-kube-api-access-494sp" pod "a99b4f25-e266-43dd-bd25-576847ed1b4a" (UID: "a99b4f25-e266-43dd-bd25-576847ed1b4a"). InnerVolumeSpecName "kube-api-access-494sp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:30.929370 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:30.919570 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a99b4f25-e266-43dd-bd25-576847ed1b4a-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:23:30.929370 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:30.919610 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-494sp\" (UniqueName: \"kubernetes.io/projected/a99b4f25-e266-43dd-bd25-576847ed1b4a-kube-api-access-494sp\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:23:30.929370 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:30.919627 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a99b4f25-e266-43dd-bd25-576847ed1b4a-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:23:30.942982 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:30.932963 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a99b4f25-e266-43dd-bd25-576847ed1b4a-metrics-certs" pod "a99b4f25-e266-43dd-bd25-576847ed1b4a" (UID: "a99b4f25-e266-43dd-bd25-576847ed1b4a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:23:31.024283 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:31.024242 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a99b4f25-e266-43dd-bd25-576847ed1b4a-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:23:31.616631 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:31.616600 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr" Oct 30 18:23:31.616805 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:31.616624 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr" event={"ID":"a99b4f25-e266-43dd-bd25-576847ed1b4a","Type":"ContainerDied","Data":"2d4f02cc4ca0ea09626888bfe50374ac152291c9c7156a3768a30bf5ea48a20e"} Oct 30 18:23:31.616805 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:31.616705 4455 scope.go:122] "RemoveContainer" containerID="415686924fb73625eeb0794e47272a3f9124bc0d7a53acca13da3c081828c726" Oct 30 18:23:31.720349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:31.716448 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr"] Oct 30 18:23:31.740828 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:31.740784 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e0bc49-67c856d48d-dhgjr"] Oct 30 18:23:33.638459 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:33.638425 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a99b4f25-e266-43dd-bd25-576847ed1b4a" path="/var/lib/kubelet/pods/a99b4f25-e266-43dd-bd25-576847ed1b4a/volumes" Oct 30 18:23:35.318995 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.318959 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b8b654-78678d94b7-nm799"] Oct 30 18:23:35.319772 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.319755 4455 cpu_manager.go:414] "containerMap: removing container" podUID="d9e693c8-eff4-4afd-8c2a-3009b6c0c27f" containerName="payload-pre-processing" Oct 30 18:23:35.319824 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.319774 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9e693c8-eff4-4afd-8c2a-3009b6c0c27f" containerName="payload-pre-processing" Oct 30 18:23:35.319824 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.319794 4455 cpu_manager.go:414] "containerMap: removing container" podUID="97e9aed5-3f20-4301-b832-a78b35c9fb71" containerName="payload-processing" Oct 30 18:23:35.319824 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.319799 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97e9aed5-3f20-4301-b832-a78b35c9fb71" containerName="payload-processing" Oct 30 18:23:35.319824 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.319812 4455 cpu_manager.go:414] "containerMap: removing container" podUID="466c7c11-2221-4749-bba1-d0f2383c66d2" containerName="maas-api" Oct 30 18:23:35.319824 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.319817 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="466c7c11-2221-4749-bba1-d0f2383c66d2" containerName="maas-api" Oct 30 18:23:35.319824 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.319826 4455 cpu_manager.go:414] "containerMap: removing container" podUID="a99b4f25-e266-43dd-bd25-576847ed1b4a" containerName="maas-api" Oct 30 18:23:35.320009 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.319831 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a99b4f25-e266-43dd-bd25-576847ed1b4a" containerName="maas-api" Oct 30 18:23:35.320009 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.319837 4455 cpu_manager.go:414] "containerMap: removing container" podUID="97e9aed5-3f20-4301-b832-a78b35c9fb71" containerName="payload-processing" Oct 30 18:23:35.320009 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.319842 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97e9aed5-3f20-4301-b832-a78b35c9fb71" containerName="payload-processing" Oct 30 18:23:35.320009 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.319950 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9e693c8-eff4-4afd-8c2a-3009b6c0c27f" containerName="payload-pre-processing" Oct 30 18:23:35.320009 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.319961 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="97e9aed5-3f20-4301-b832-a78b35c9fb71" containerName="payload-processing" Oct 30 18:23:35.320009 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.319968 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="466c7c11-2221-4749-bba1-d0f2383c66d2" containerName="maas-api" Oct 30 18:23:35.320009 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.319975 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="a99b4f25-e266-43dd-bd25-576847ed1b4a" containerName="maas-api" Oct 30 18:23:35.320009 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.319981 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="97e9aed5-3f20-4301-b832-a78b35c9fb71" containerName="payload-processing" Oct 30 18:23:35.325031 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.325013 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8b654-78678d94b7-nm799" Oct 30 18:23:35.346606 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.346574 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b8b654-78678d94b7-nm799"] Oct 30 18:23:35.353739 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.353714 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-b8b654\"" Oct 30 18:23:35.353739 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.353718 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-b8b654-dockercfg-mwsld\"" Oct 30 18:23:35.551346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.480724 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/705d36c4-b030-4767-86f8-351552d25d53-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b8b654-78678d94b7-nm799\" (UID: \"705d36c4-b030-4767-86f8-351552d25d53\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8b654-78678d94b7-nm799" Oct 30 18:23:35.625851 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.599919 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b8b654-6c7556f4cb-8prx6"] Oct 30 18:23:35.639233 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.625880 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdktv\" (UniqueName: \"kubernetes.io/projected/705d36c4-b030-4767-86f8-351552d25d53-kube-api-access-hdktv\") pod \"payload-pre-processing-e2e-praxis-b-b8b654-78678d94b7-nm799\" (UID: \"705d36c4-b030-4767-86f8-351552d25d53\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8b654-78678d94b7-nm799" Oct 30 18:23:35.639741 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.639718 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b8b654-6c7556f4cb-8prx6" Oct 30 18:23:35.709342 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.709284 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b8b654-6c7556f4cb-8prx6"] Oct 30 18:23:35.740997 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.740959 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdktv\" (UniqueName: \"kubernetes.io/projected/705d36c4-b030-4767-86f8-351552d25d53-kube-api-access-hdktv\") pod \"payload-pre-processing-e2e-praxis-b-b8b654-78678d94b7-nm799\" (UID: \"705d36c4-b030-4767-86f8-351552d25d53\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8b654-78678d94b7-nm799" Oct 30 18:23:35.741207 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.741144 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzxkj\" (UniqueName: \"kubernetes.io/projected/9e68229e-2581-4c3b-8ab5-e1c867973ac1-kube-api-access-gzxkj\") pod \"payload-processing-e2e-praxis-b-b8b654-6c7556f4cb-8prx6\" (UID: \"9e68229e-2581-4c3b-8ab5-e1c867973ac1\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b8b654-6c7556f4cb-8prx6" Oct 30 18:23:35.741431 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.741410 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/705d36c4-b030-4767-86f8-351552d25d53-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b8b654-78678d94b7-nm799\" (UID: \"705d36c4-b030-4767-86f8-351552d25d53\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8b654-78678d94b7-nm799" Oct 30 18:23:35.741519 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.741488 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e68229e-2581-4c3b-8ab5-e1c867973ac1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b8b654-6c7556f4cb-8prx6\" (UID: \"9e68229e-2581-4c3b-8ab5-e1c867973ac1\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b8b654-6c7556f4cb-8prx6" Oct 30 18:23:35.742083 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.742062 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/705d36c4-b030-4767-86f8-351552d25d53-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b8b654-78678d94b7-nm799\" (UID: \"705d36c4-b030-4767-86f8-351552d25d53\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8b654-78678d94b7-nm799" Oct 30 18:23:35.766072 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.766044 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdktv\" (UniqueName: \"kubernetes.io/projected/705d36c4-b030-4767-86f8-351552d25d53-kube-api-access-hdktv\") pod \"payload-pre-processing-e2e-praxis-b-b8b654-78678d94b7-nm799\" (UID: \"705d36c4-b030-4767-86f8-351552d25d53\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8b654-78678d94b7-nm799" Oct 30 18:23:35.842902 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.842851 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e68229e-2581-4c3b-8ab5-e1c867973ac1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b8b654-6c7556f4cb-8prx6\" (UID: \"9e68229e-2581-4c3b-8ab5-e1c867973ac1\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b8b654-6c7556f4cb-8prx6" Oct 30 18:23:35.843806 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.843774 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e68229e-2581-4c3b-8ab5-e1c867973ac1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b8b654-6c7556f4cb-8prx6\" (UID: \"9e68229e-2581-4c3b-8ab5-e1c867973ac1\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b8b654-6c7556f4cb-8prx6" Oct 30 18:23:35.844997 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.844026 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzxkj\" (UniqueName: \"kubernetes.io/projected/9e68229e-2581-4c3b-8ab5-e1c867973ac1-kube-api-access-gzxkj\") pod \"payload-processing-e2e-praxis-b-b8b654-6c7556f4cb-8prx6\" (UID: \"9e68229e-2581-4c3b-8ab5-e1c867973ac1\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b8b654-6c7556f4cb-8prx6" Oct 30 18:23:35.869548 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.869522 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzxkj\" (UniqueName: \"kubernetes.io/projected/9e68229e-2581-4c3b-8ab5-e1c867973ac1-kube-api-access-gzxkj\") pod \"payload-processing-e2e-praxis-b-b8b654-6c7556f4cb-8prx6\" (UID: \"9e68229e-2581-4c3b-8ab5-e1c867973ac1\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b8b654-6c7556f4cb-8prx6" Oct 30 18:23:35.940614 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:35.940536 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8b654-78678d94b7-nm799" Oct 30 18:23:36.030228 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:36.030086 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b8b654-6c7556f4cb-8prx6" Oct 30 18:23:36.281175 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:36.281112 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b8b654-78678d94b7-nm799"] Oct 30 18:23:36.322350 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:23:36.322242 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod705d36c4_b030_4767_86f8_351552d25d53.slice/crio-8605e69ddab8c399c4d66cae58e740b43803c14aa71d7f5c92985b27c00dde94 WatchSource:0}: Error finding container 8605e69ddab8c399c4d66cae58e740b43803c14aa71d7f5c92985b27c00dde94: Status 404 returned error can't find the container with id 8605e69ddab8c399c4d66cae58e740b43803c14aa71d7f5c92985b27c00dde94 Oct 30 18:23:36.391605 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:36.391575 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b8b654-6c7556f4cb-8prx6"] Oct 30 18:23:36.395494 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:23:36.395161 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e68229e_2581_4c3b_8ab5_e1c867973ac1.slice/crio-6efe387f4bc4717cf9a20c1a48f61122d5b56f95c8f4eb0bf93494a5517fe817 WatchSource:0}: Error finding container 6efe387f4bc4717cf9a20c1a48f61122d5b56f95c8f4eb0bf93494a5517fe817: Status 404 returned error can't find the container with id 6efe387f4bc4717cf9a20c1a48f61122d5b56f95c8f4eb0bf93494a5517fe817 Oct 30 18:23:36.771826 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:36.771787 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8b654-78678d94b7-nm799" event={"ID":"705d36c4-b030-4767-86f8-351552d25d53","Type":"ContainerStarted","Data":"f4eec0b858ace8269d056683ad0642b4fb90cdce65a1d7a337aaea876f139d99"} Oct 30 18:23:36.772101 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:36.772076 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8b654-78678d94b7-nm799" event={"ID":"705d36c4-b030-4767-86f8-351552d25d53","Type":"ContainerStarted","Data":"8605e69ddab8c399c4d66cae58e740b43803c14aa71d7f5c92985b27c00dde94"} Oct 30 18:23:36.779501 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:36.779471 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8b654-78678d94b7-nm799" Oct 30 18:23:36.792867 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:36.792794 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b8b654-6c7556f4cb-8prx6" event={"ID":"9e68229e-2581-4c3b-8ab5-e1c867973ac1","Type":"ContainerStarted","Data":"9475b7b587ac109c5bdace08cc733f99fdcfb779e966646b3d0bffae4b67280e"} Oct 30 18:23:36.793199 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:36.793151 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b8b654-6c7556f4cb-8prx6" event={"ID":"9e68229e-2581-4c3b-8ab5-e1c867973ac1","Type":"ContainerStarted","Data":"6efe387f4bc4717cf9a20c1a48f61122d5b56f95c8f4eb0bf93494a5517fe817"} Oct 30 18:23:36.794389 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:36.794365 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-b8b654-6c7556f4cb-8prx6" Oct 30 18:23:36.842627 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:36.842581 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8b654-78678d94b7-nm799" podStartSLOduration=1.842566871 podStartE2EDuration="1.842566871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:36.840517933 +0000 UTC m=+1869.328154967" watchObservedRunningTime="2026-10-30 18:23:36.842566871 +0000 UTC m=+1869.330203904" Oct 30 18:23:45.014576 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:45.014528 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:47.135483 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:47.135429 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:48.083823 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:48.083062 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-b8b654-6c7556f4cb-8prx6" podStartSLOduration=13.083041213 podStartE2EDuration="13.083041213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:36.884945192 +0000 UTC m=+1869.372582231" watchObservedRunningTime="2026-10-30 18:23:48.083041213 +0000 UTC m=+1880.570678249" Oct 30 18:23:48.147719 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:48.147683 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7ffac8-fd6b99659-kp2l5"] Oct 30 18:23:48.269917 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:48.269880 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7ffac8-fd6b99659-kp2l5" Oct 30 18:23:48.306723 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:48.299204 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-7ffac8\"" Oct 30 18:23:48.311836 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:48.308847 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-7ffac8-dockercfg-njh6t\"" Oct 30 18:23:48.311836 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:48.310434 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7ffac8-fd6b99659-kp2l5"] Oct 30 18:23:48.380749 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:48.380654 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7ffac8-6d59c957dd-qplqj"] Oct 30 18:23:48.456360 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:48.452961 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25wx2\" (UniqueName: \"kubernetes.io/projected/73040eeb-fa91-44c3-9b4f-70fe18fcdaab-kube-api-access-25wx2\") pod \"payload-pre-processing-e2e-praxis-b-7ffac8-fd6b99659-kp2l5\" (UID: \"73040eeb-fa91-44c3-9b4f-70fe18fcdaab\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7ffac8-fd6b99659-kp2l5" Oct 30 18:23:48.456360 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:48.453176 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73040eeb-fa91-44c3-9b4f-70fe18fcdaab-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7ffac8-fd6b99659-kp2l5\" (UID: \"73040eeb-fa91-44c3-9b4f-70fe18fcdaab\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7ffac8-fd6b99659-kp2l5" Oct 30 18:23:48.456360 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:48.453512 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7ffac8-6d59c957dd-qplqj"] Oct 30 18:23:48.456360 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:48.453649 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7ffac8-6d59c957dd-qplqj" Oct 30 18:23:48.557256 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:48.556973 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25wx2\" (UniqueName: \"kubernetes.io/projected/73040eeb-fa91-44c3-9b4f-70fe18fcdaab-kube-api-access-25wx2\") pod \"payload-pre-processing-e2e-praxis-b-7ffac8-fd6b99659-kp2l5\" (UID: \"73040eeb-fa91-44c3-9b4f-70fe18fcdaab\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7ffac8-fd6b99659-kp2l5" Oct 30 18:23:48.558354 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:48.557204 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73040eeb-fa91-44c3-9b4f-70fe18fcdaab-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7ffac8-fd6b99659-kp2l5\" (UID: \"73040eeb-fa91-44c3-9b4f-70fe18fcdaab\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7ffac8-fd6b99659-kp2l5" Oct 30 18:23:48.558354 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:48.557648 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvmc9\" (UniqueName: \"kubernetes.io/projected/a754cbb1-3b1e-499d-b9e2-1a8445b4fd83-kube-api-access-wvmc9\") pod \"payload-processing-e2e-praxis-b-7ffac8-6d59c957dd-qplqj\" (UID: \"a754cbb1-3b1e-499d-b9e2-1a8445b4fd83\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7ffac8-6d59c957dd-qplqj" Oct 30 18:23:48.558354 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:48.557719 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a754cbb1-3b1e-499d-b9e2-1a8445b4fd83-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7ffac8-6d59c957dd-qplqj\" (UID: \"a754cbb1-3b1e-499d-b9e2-1a8445b4fd83\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7ffac8-6d59c957dd-qplqj" Oct 30 18:23:48.559394 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:48.559370 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73040eeb-fa91-44c3-9b4f-70fe18fcdaab-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7ffac8-fd6b99659-kp2l5\" (UID: \"73040eeb-fa91-44c3-9b4f-70fe18fcdaab\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7ffac8-fd6b99659-kp2l5" Oct 30 18:23:48.654446 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:48.654360 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25wx2\" (UniqueName: \"kubernetes.io/projected/73040eeb-fa91-44c3-9b4f-70fe18fcdaab-kube-api-access-25wx2\") pod \"payload-pre-processing-e2e-praxis-b-7ffac8-fd6b99659-kp2l5\" (UID: \"73040eeb-fa91-44c3-9b4f-70fe18fcdaab\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7ffac8-fd6b99659-kp2l5" Oct 30 18:23:48.665683 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:48.661597 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvmc9\" (UniqueName: \"kubernetes.io/projected/a754cbb1-3b1e-499d-b9e2-1a8445b4fd83-kube-api-access-wvmc9\") pod \"payload-processing-e2e-praxis-b-7ffac8-6d59c957dd-qplqj\" (UID: \"a754cbb1-3b1e-499d-b9e2-1a8445b4fd83\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7ffac8-6d59c957dd-qplqj" Oct 30 18:23:48.665892 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:48.665782 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a754cbb1-3b1e-499d-b9e2-1a8445b4fd83-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7ffac8-6d59c957dd-qplqj\" (UID: \"a754cbb1-3b1e-499d-b9e2-1a8445b4fd83\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7ffac8-6d59c957dd-qplqj" Oct 30 18:23:48.667035 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:48.667003 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a754cbb1-3b1e-499d-b9e2-1a8445b4fd83-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7ffac8-6d59c957dd-qplqj\" (UID: \"a754cbb1-3b1e-499d-b9e2-1a8445b4fd83\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7ffac8-6d59c957dd-qplqj" Oct 30 18:23:48.699655 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:48.699622 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvmc9\" (UniqueName: \"kubernetes.io/projected/a754cbb1-3b1e-499d-b9e2-1a8445b4fd83-kube-api-access-wvmc9\") pod \"payload-processing-e2e-praxis-b-7ffac8-6d59c957dd-qplqj\" (UID: \"a754cbb1-3b1e-499d-b9e2-1a8445b4fd83\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7ffac8-6d59c957dd-qplqj" Oct 30 18:23:48.834283 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:48.834248 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7ffac8-fd6b99659-kp2l5" Oct 30 18:23:48.853305 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:48.853278 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7ffac8-6d59c957dd-qplqj" Oct 30 18:23:48.879421 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:48.878629 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8b654-78678d94b7-nm799" Oct 30 18:23:48.882233 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:48.882211 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-b8b654-6c7556f4cb-8prx6" Oct 30 18:23:49.219869 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:49.219838 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7ffac8-fd6b99659-kp2l5"] Oct 30 18:23:49.285168 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:49.271736 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7ffac8-6d59c957dd-qplqj"] Oct 30 18:23:49.797001 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:49.796962 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7ffac8-fd6b99659-kp2l5" event={"ID":"73040eeb-fa91-44c3-9b4f-70fe18fcdaab","Type":"ContainerStarted","Data":"ace5220c593576a0bb7a962ee389a893ba15fa6bf67feebaafb227d3ed268bff"} Oct 30 18:23:49.797306 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:49.797265 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7ffac8-fd6b99659-kp2l5" event={"ID":"73040eeb-fa91-44c3-9b4f-70fe18fcdaab","Type":"ContainerStarted","Data":"dad477254cdeb450910c1f9a82d920b19bc3039ae91cdf3fe2c87487663f97fe"} Oct 30 18:23:49.797889 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:49.797868 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7ffac8-fd6b99659-kp2l5" Oct 30 18:23:49.801268 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:49.801242 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7ffac8-6d59c957dd-qplqj" event={"ID":"a754cbb1-3b1e-499d-b9e2-1a8445b4fd83","Type":"ContainerStarted","Data":"376a9fe37ae8e11d1ca887485169eb801f1a87fadf2519c8cd1f649eedd23060"} Oct 30 18:23:49.801427 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:49.801288 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7ffac8-6d59c957dd-qplqj" event={"ID":"a754cbb1-3b1e-499d-b9e2-1a8445b4fd83","Type":"ContainerStarted","Data":"9a8cfa878a8f58f75957b5bcc0fdf64b3b70a34f008ba7a028f462c3bb307266"} Oct 30 18:23:49.802044 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:49.802015 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-7ffac8-6d59c957dd-qplqj" Oct 30 18:23:49.849573 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:49.849519 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7ffac8-fd6b99659-kp2l5" podStartSLOduration=2.849504015 podStartE2EDuration="2.849504015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:49.845452799 +0000 UTC m=+1882.333089832" watchObservedRunningTime="2026-10-30 18:23:49.849504015 +0000 UTC m=+1882.337141048" Oct 30 18:23:49.894406 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:49.894355 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-7ffac8-6d59c957dd-qplqj" podStartSLOduration=2.894341181 podStartE2EDuration="2.894341181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:49.891651277 +0000 UTC m=+1882.379288312" watchObservedRunningTime="2026-10-30 18:23:49.894341181 +0000 UTC m=+1882.381978206" Oct 30 18:23:56.256459 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:56.256422 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq"] Oct 30 18:23:56.261304 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:56.261279 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq" Oct 30 18:23:56.284440 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:56.284410 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq"] Oct 30 18:23:56.297370 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:56.296406 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-b8b654-serving-cert\"" Oct 30 18:23:56.305704 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:56.305671 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d2b2365-0487-4dac-8482-eb697a969a09-metrics-certs\") pod \"maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq\" (UID: \"8d2b2365-0487-4dac-8482-eb697a969a09\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq" Oct 30 18:23:56.305871 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:56.305752 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d2b2365-0487-4dac-8482-eb697a969a09-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq\" (UID: \"8d2b2365-0487-4dac-8482-eb697a969a09\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq" Oct 30 18:23:56.305871 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:56.305783 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qklvq\" (UniqueName: \"kubernetes.io/projected/8d2b2365-0487-4dac-8482-eb697a969a09-kube-api-access-qklvq\") pod \"maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq\" (UID: \"8d2b2365-0487-4dac-8482-eb697a969a09\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq" Oct 30 18:23:56.306367 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:56.306027 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d2b2365-0487-4dac-8482-eb697a969a09-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq\" (UID: \"8d2b2365-0487-4dac-8482-eb697a969a09\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq" Oct 30 18:23:56.407480 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:56.407447 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d2b2365-0487-4dac-8482-eb697a969a09-metrics-certs\") pod \"maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq\" (UID: \"8d2b2365-0487-4dac-8482-eb697a969a09\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq" Oct 30 18:23:56.407668 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:56.407508 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d2b2365-0487-4dac-8482-eb697a969a09-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq\" (UID: \"8d2b2365-0487-4dac-8482-eb697a969a09\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq" Oct 30 18:23:56.407668 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:56.407537 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qklvq\" (UniqueName: \"kubernetes.io/projected/8d2b2365-0487-4dac-8482-eb697a969a09-kube-api-access-qklvq\") pod \"maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq\" (UID: \"8d2b2365-0487-4dac-8482-eb697a969a09\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq" Oct 30 18:23:56.407786 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:56.407747 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d2b2365-0487-4dac-8482-eb697a969a09-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq\" (UID: \"8d2b2365-0487-4dac-8482-eb697a969a09\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq" Oct 30 18:23:56.408770 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:56.408741 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d2b2365-0487-4dac-8482-eb697a969a09-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq\" (UID: \"8d2b2365-0487-4dac-8482-eb697a969a09\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq" Oct 30 18:23:56.410367 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:56.410306 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d2b2365-0487-4dac-8482-eb697a969a09-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq\" (UID: \"8d2b2365-0487-4dac-8482-eb697a969a09\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq" Oct 30 18:23:56.410739 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:56.410714 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d2b2365-0487-4dac-8482-eb697a969a09-metrics-certs\") pod \"maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq\" (UID: \"8d2b2365-0487-4dac-8482-eb697a969a09\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq" Oct 30 18:23:56.444415 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:56.444388 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qklvq\" (UniqueName: \"kubernetes.io/projected/8d2b2365-0487-4dac-8482-eb697a969a09-kube-api-access-qklvq\") pod \"maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq\" (UID: \"8d2b2365-0487-4dac-8482-eb697a969a09\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq" Oct 30 18:23:56.579576 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:56.578663 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq" Oct 30 18:23:56.903049 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:23:56.903014 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d2b2365_0487_4dac_8482_eb697a969a09.slice/crio-b366109fa502396fe06b1686fc05f0a943680745c2a65f107047f2424d310e55 WatchSource:0}: Error finding container b366109fa502396fe06b1686fc05f0a943680745c2a65f107047f2424d310e55: Status 404 returned error can't find the container with id b366109fa502396fe06b1686fc05f0a943680745c2a65f107047f2424d310e55 Oct 30 18:23:56.904528 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:56.904461 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq"] Oct 30 18:23:57.659339 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:57.659228 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1fae38-6b77fbf575-d99xj"] Oct 30 18:23:57.665165 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:57.665132 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1fae38-6b77fbf575-d99xj" Oct 30 18:23:57.705419 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:57.703810 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-1fae38\"" Oct 30 18:23:57.706257 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:57.706103 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1fae38-6b77fbf575-d99xj"] Oct 30 18:23:57.717822 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:57.717794 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-1fae38-dockercfg-4h469\"" Oct 30 18:23:57.750706 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:57.749041 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d14b8e74-04c8-4948-8e26-12c06bc39341-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-1fae38-6b77fbf575-d99xj\" (UID: \"d14b8e74-04c8-4948-8e26-12c06bc39341\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1fae38-6b77fbf575-d99xj" Oct 30 18:23:57.750706 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:57.749239 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckp97\" (UniqueName: \"kubernetes.io/projected/d14b8e74-04c8-4948-8e26-12c06bc39341-kube-api-access-ckp97\") pod \"payload-pre-processing-e2e-praxis-a-1fae38-6b77fbf575-d99xj\" (UID: \"d14b8e74-04c8-4948-8e26-12c06bc39341\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1fae38-6b77fbf575-d99xj" Oct 30 18:23:57.892362 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:57.850396 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckp97\" (UniqueName: \"kubernetes.io/projected/d14b8e74-04c8-4948-8e26-12c06bc39341-kube-api-access-ckp97\") pod \"payload-pre-processing-e2e-praxis-a-1fae38-6b77fbf575-d99xj\" (UID: \"d14b8e74-04c8-4948-8e26-12c06bc39341\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1fae38-6b77fbf575-d99xj" Oct 30 18:23:58.062656 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:58.061633 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d14b8e74-04c8-4948-8e26-12c06bc39341-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-1fae38-6b77fbf575-d99xj\" (UID: \"d14b8e74-04c8-4948-8e26-12c06bc39341\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1fae38-6b77fbf575-d99xj" Oct 30 18:23:58.068665 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:58.068629 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d14b8e74-04c8-4948-8e26-12c06bc39341-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-1fae38-6b77fbf575-d99xj\" (UID: \"d14b8e74-04c8-4948-8e26-12c06bc39341\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1fae38-6b77fbf575-d99xj" Oct 30 18:23:58.078617 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:58.078583 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1fae38-d874c668-m42lh"] Oct 30 18:23:58.133560 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:58.133521 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckp97\" (UniqueName: \"kubernetes.io/projected/d14b8e74-04c8-4948-8e26-12c06bc39341-kube-api-access-ckp97\") pod \"payload-pre-processing-e2e-praxis-a-1fae38-6b77fbf575-d99xj\" (UID: \"d14b8e74-04c8-4948-8e26-12c06bc39341\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1fae38-6b77fbf575-d99xj" Oct 30 18:23:58.160105 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:58.160057 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq" event={"ID":"8d2b2365-0487-4dac-8482-eb697a969a09","Type":"ContainerStarted","Data":"886d3b49e1390f308b9bfaead68776952bb82e677b9bcb4b484cea00bd94d6d5"} Oct 30 18:23:58.160307 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:58.160137 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1fae38-d874c668-m42lh"] Oct 30 18:23:58.160307 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:58.160215 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq" Oct 30 18:23:58.160307 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:58.160276 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq" event={"ID":"8d2b2365-0487-4dac-8482-eb697a969a09","Type":"ContainerStarted","Data":"b366109fa502396fe06b1686fc05f0a943680745c2a65f107047f2424d310e55"} Oct 30 18:23:58.161375 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:58.161263 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1fae38-d874c668-m42lh" Oct 30 18:23:58.172795 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:58.172734 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq" podStartSLOduration=1.819243003 podStartE2EDuration="2.172712749s" totalImagesPullingTime="353.469746ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:58.135753259 +0000 UTC m=+1890.623390320" watchObservedRunningTime="2026-10-30 18:23:58.172712749 +0000 UTC m=+1890.660349808" Oct 30 18:23:58.281352 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:58.279371 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flbng\" (UniqueName: \"kubernetes.io/projected/c436ce58-c966-4069-bbe3-747b16908679-kube-api-access-flbng\") pod \"payload-processing-e2e-praxis-a-1fae38-d874c668-m42lh\" (UID: \"c436ce58-c966-4069-bbe3-747b16908679\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1fae38-d874c668-m42lh" Oct 30 18:23:58.281352 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:58.279572 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c436ce58-c966-4069-bbe3-747b16908679-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-1fae38-d874c668-m42lh\" (UID: \"c436ce58-c966-4069-bbe3-747b16908679\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1fae38-d874c668-m42lh" Oct 30 18:23:58.317940 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:58.317851 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1fae38-6b77fbf575-d99xj" Oct 30 18:23:58.382138 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:58.382088 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flbng\" (UniqueName: \"kubernetes.io/projected/c436ce58-c966-4069-bbe3-747b16908679-kube-api-access-flbng\") pod \"payload-processing-e2e-praxis-a-1fae38-d874c668-m42lh\" (UID: \"c436ce58-c966-4069-bbe3-747b16908679\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1fae38-d874c668-m42lh" Oct 30 18:23:58.382436 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:58.382296 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c436ce58-c966-4069-bbe3-747b16908679-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-1fae38-d874c668-m42lh\" (UID: \"c436ce58-c966-4069-bbe3-747b16908679\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1fae38-d874c668-m42lh" Oct 30 18:23:58.383630 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:58.383604 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c436ce58-c966-4069-bbe3-747b16908679-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-1fae38-d874c668-m42lh\" (UID: \"c436ce58-c966-4069-bbe3-747b16908679\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1fae38-d874c668-m42lh" Oct 30 18:23:58.410050 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:58.410018 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flbng\" (UniqueName: \"kubernetes.io/projected/c436ce58-c966-4069-bbe3-747b16908679-kube-api-access-flbng\") pod \"payload-processing-e2e-praxis-a-1fae38-d874c668-m42lh\" (UID: \"c436ce58-c966-4069-bbe3-747b16908679\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1fae38-d874c668-m42lh" Oct 30 18:23:58.505659 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:58.505386 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1fae38-d874c668-m42lh" Oct 30 18:23:58.751422 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:58.751395 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1fae38-6b77fbf575-d99xj"] Oct 30 18:23:59.104438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:59.088166 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1fae38-d874c668-m42lh"] Oct 30 18:23:59.147888 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:59.145596 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1fae38-6b77fbf575-d99xj" event={"ID":"d14b8e74-04c8-4948-8e26-12c06bc39341","Type":"ContainerStarted","Data":"8261d8c77fb921026e092e09fc902eaa0bbdb74b86ff7371d0bd2ba03005e6dc"} Oct 30 18:23:59.147888 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:59.145655 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1fae38-6b77fbf575-d99xj" event={"ID":"d14b8e74-04c8-4948-8e26-12c06bc39341","Type":"ContainerStarted","Data":"79ca4e3cb145ba86ecd998872533635904d71681b4707141e30b9f4fd12c60b9"} Oct 30 18:23:59.147888 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:59.147460 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1fae38-6b77fbf575-d99xj" Oct 30 18:23:59.211347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:23:59.207481 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1fae38-6b77fbf575-d99xj" podStartSLOduration=2.207463055 podStartE2EDuration="2.207463055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:59.198671023 +0000 UTC m=+1891.686308058" watchObservedRunningTime="2026-10-30 18:23:59.207463055 +0000 UTC m=+1891.695100090" Oct 30 18:24:00.167550 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:00.167490 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1fae38-d874c668-m42lh" event={"ID":"c436ce58-c966-4069-bbe3-747b16908679","Type":"ContainerStarted","Data":"20b958598c2497793f2163aaf8afa2c2cbe56a1cca952b928f3a5468ebb81312"} Oct 30 18:24:00.168010 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:00.167612 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1fae38-d874c668-m42lh" event={"ID":"c436ce58-c966-4069-bbe3-747b16908679","Type":"ContainerStarted","Data":"75bab5d1874cb0b0a02a54e2e595961ac303ee276dd9e9a35885c3851351af40"} Oct 30 18:24:00.168426 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:00.168402 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-1fae38-d874c668-m42lh" Oct 30 18:24:00.213771 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:00.213716 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-1fae38-d874c668-m42lh" podStartSLOduration=3.213703368 podStartE2EDuration="3.213703368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:00.213231488 +0000 UTC m=+1892.700868524" watchObservedRunningTime="2026-10-30 18:24:00.213703368 +0000 UTC m=+1892.701340423" Oct 30 18:24:00.443100 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:24:00.443005 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:24:01.823358 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:01.822704 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7ffac8-fd6b99659-kp2l5" Oct 30 18:24:01.823358 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:01.823202 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-7ffac8-6d59c957dd-qplqj" Oct 30 18:24:05.174480 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:05.174451 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq" Oct 30 18:24:07.153040 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:07.153005 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69"] Oct 30 18:24:07.159258 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:07.159237 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69" Oct 30 18:24:07.193220 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:07.192536 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-7ffac8-serving-cert\"" Oct 30 18:24:07.198927 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:07.198831 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69"] Oct 30 18:24:07.258428 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:07.258385 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80b01b7a-2c9f-41b5-ba92-9ade95a89cc8-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69\" (UID: \"80b01b7a-2c9f-41b5-ba92-9ade95a89cc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69" Oct 30 18:24:07.258617 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:07.258444 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80b01b7a-2c9f-41b5-ba92-9ade95a89cc8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69\" (UID: \"80b01b7a-2c9f-41b5-ba92-9ade95a89cc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69" Oct 30 18:24:07.258617 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:07.258607 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrrw4\" (UniqueName: \"kubernetes.io/projected/80b01b7a-2c9f-41b5-ba92-9ade95a89cc8-kube-api-access-qrrw4\") pod \"maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69\" (UID: \"80b01b7a-2c9f-41b5-ba92-9ade95a89cc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69" Oct 30 18:24:07.258744 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:07.258640 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b01b7a-2c9f-41b5-ba92-9ade95a89cc8-metrics-certs\") pod \"maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69\" (UID: \"80b01b7a-2c9f-41b5-ba92-9ade95a89cc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69" Oct 30 18:24:07.361345 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:07.360591 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80b01b7a-2c9f-41b5-ba92-9ade95a89cc8-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69\" (UID: \"80b01b7a-2c9f-41b5-ba92-9ade95a89cc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69" Oct 30 18:24:07.361345 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:07.360650 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80b01b7a-2c9f-41b5-ba92-9ade95a89cc8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69\" (UID: \"80b01b7a-2c9f-41b5-ba92-9ade95a89cc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69" Oct 30 18:24:07.361345 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:07.360790 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrrw4\" (UniqueName: \"kubernetes.io/projected/80b01b7a-2c9f-41b5-ba92-9ade95a89cc8-kube-api-access-qrrw4\") pod \"maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69\" (UID: \"80b01b7a-2c9f-41b5-ba92-9ade95a89cc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69" Oct 30 18:24:07.361345 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:07.360843 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b01b7a-2c9f-41b5-ba92-9ade95a89cc8-metrics-certs\") pod \"maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69\" (UID: \"80b01b7a-2c9f-41b5-ba92-9ade95a89cc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69" Oct 30 18:24:07.368719 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:07.367399 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80b01b7a-2c9f-41b5-ba92-9ade95a89cc8-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69\" (UID: \"80b01b7a-2c9f-41b5-ba92-9ade95a89cc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69" Oct 30 18:24:07.393746 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:07.393690 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrrw4\" (UniqueName: \"kubernetes.io/projected/80b01b7a-2c9f-41b5-ba92-9ade95a89cc8-kube-api-access-qrrw4\") pod \"maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69\" (UID: \"80b01b7a-2c9f-41b5-ba92-9ade95a89cc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69" Oct 30 18:24:07.416232 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:07.414601 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80b01b7a-2c9f-41b5-ba92-9ade95a89cc8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69\" (UID: \"80b01b7a-2c9f-41b5-ba92-9ade95a89cc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69" Oct 30 18:24:07.416232 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:07.415534 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b01b7a-2c9f-41b5-ba92-9ade95a89cc8-metrics-certs\") pod \"maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69\" (UID: \"80b01b7a-2c9f-41b5-ba92-9ade95a89cc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69" Oct 30 18:24:07.474699 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:07.474649 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69" Oct 30 18:24:07.800710 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:24:07.800679 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b01b7a_2c9f_41b5_ba92_9ade95a89cc8.slice/crio-2b5d039e5ead39731678cf31b3e0ef3c0bad75dbf31b3617655f27817a659d4f WatchSource:0}: Error finding container 2b5d039e5ead39731678cf31b3e0ef3c0bad75dbf31b3617655f27817a659d4f: Status 404 returned error can't find the container with id 2b5d039e5ead39731678cf31b3e0ef3c0bad75dbf31b3617655f27817a659d4f Oct 30 18:24:07.802177 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:07.802153 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69"] Oct 30 18:24:08.442692 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:08.442591 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69" event={"ID":"80b01b7a-2c9f-41b5-ba92-9ade95a89cc8","Type":"ContainerStarted","Data":"8af70fe921a366f858d2bfde039bf60ee64e59652dee0cafdee698b46e80551f"} Oct 30 18:24:08.443110 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:08.442703 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69" event={"ID":"80b01b7a-2c9f-41b5-ba92-9ade95a89cc8","Type":"ContainerStarted","Data":"2b5d039e5ead39731678cf31b3e0ef3c0bad75dbf31b3617655f27817a659d4f"} Oct 30 18:24:08.443110 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:08.442770 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69" Oct 30 18:24:08.494970 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:08.494901 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69" podStartSLOduration=1.173945169 podStartE2EDuration="1.494883233s" totalImagesPullingTime="320.938064ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:08.488134453 +0000 UTC m=+1900.975771488" watchObservedRunningTime="2026-10-30 18:24:08.494883233 +0000 UTC m=+1900.982520267" Oct 30 18:24:11.182777 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:11.182744 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1fae38-6b77fbf575-d99xj" Oct 30 18:24:11.550616 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:11.550582 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-546655d4c5-knrgr"] Oct 30 18:24:11.555151 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:11.555131 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-546655d4c5-knrgr" Oct 30 18:24:11.581180 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:11.581145 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-546655d4c5-knrgr"] Oct 30 18:24:11.581507 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:11.581489 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-77db82\"" Oct 30 18:24:11.582087 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:11.582056 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-77db82-dockercfg-vz95q\"" Oct 30 18:24:11.910582 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:11.907517 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-77db82-54bd486f48-6dv4g"] Oct 30 18:24:11.933572 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:11.933517 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-77db82-54bd486f48-6dv4g"] Oct 30 18:24:11.933785 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:11.933664 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-54bd486f48-6dv4g" Oct 30 18:24:11.991378 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:11.991307 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwtsd\" (UniqueName: \"kubernetes.io/projected/2f5acd91-9868-400b-b97f-f5416477f590-kube-api-access-qwtsd\") pod \"payload-pre-processing-e2e-ipp-a-77db82-546655d4c5-knrgr\" (UID: \"2f5acd91-9868-400b-b97f-f5416477f590\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-546655d4c5-knrgr" Oct 30 18:24:11.991857 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:11.991830 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5acd91-9868-400b-b97f-f5416477f590-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-77db82-546655d4c5-knrgr\" (UID: \"2f5acd91-9868-400b-b97f-f5416477f590\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-546655d4c5-knrgr" Oct 30 18:24:12.093182 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:12.093137 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5acd91-9868-400b-b97f-f5416477f590-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-77db82-546655d4c5-knrgr\" (UID: \"2f5acd91-9868-400b-b97f-f5416477f590\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-546655d4c5-knrgr" Oct 30 18:24:12.093454 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:12.093249 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwtsd\" (UniqueName: \"kubernetes.io/projected/2f5acd91-9868-400b-b97f-f5416477f590-kube-api-access-qwtsd\") pod \"payload-pre-processing-e2e-ipp-a-77db82-546655d4c5-knrgr\" (UID: \"2f5acd91-9868-400b-b97f-f5416477f590\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-546655d4c5-knrgr" Oct 30 18:24:12.093454 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:12.093384 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6lkt\" (UniqueName: \"kubernetes.io/projected/30592a4b-d20f-44ab-bb1d-7bd5aaf9163d-kube-api-access-j6lkt\") pod \"payload-processing-e2e-ipp-a-77db82-54bd486f48-6dv4g\" (UID: \"30592a4b-d20f-44ab-bb1d-7bd5aaf9163d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-54bd486f48-6dv4g" Oct 30 18:24:12.093636 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:12.093494 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30592a4b-d20f-44ab-bb1d-7bd5aaf9163d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-77db82-54bd486f48-6dv4g\" (UID: \"30592a4b-d20f-44ab-bb1d-7bd5aaf9163d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-54bd486f48-6dv4g" Oct 30 18:24:12.094400 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:12.094371 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5acd91-9868-400b-b97f-f5416477f590-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-77db82-546655d4c5-knrgr\" (UID: \"2f5acd91-9868-400b-b97f-f5416477f590\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-546655d4c5-knrgr" Oct 30 18:24:12.118950 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:12.118918 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwtsd\" (UniqueName: \"kubernetes.io/projected/2f5acd91-9868-400b-b97f-f5416477f590-kube-api-access-qwtsd\") pod \"payload-pre-processing-e2e-ipp-a-77db82-546655d4c5-knrgr\" (UID: \"2f5acd91-9868-400b-b97f-f5416477f590\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-546655d4c5-knrgr" Oct 30 18:24:12.172094 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:12.172015 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-546655d4c5-knrgr" Oct 30 18:24:12.195544 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:12.195493 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6lkt\" (UniqueName: \"kubernetes.io/projected/30592a4b-d20f-44ab-bb1d-7bd5aaf9163d-kube-api-access-j6lkt\") pod \"payload-processing-e2e-ipp-a-77db82-54bd486f48-6dv4g\" (UID: \"30592a4b-d20f-44ab-bb1d-7bd5aaf9163d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-54bd486f48-6dv4g" Oct 30 18:24:12.195998 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:12.195646 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30592a4b-d20f-44ab-bb1d-7bd5aaf9163d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-77db82-54bd486f48-6dv4g\" (UID: \"30592a4b-d20f-44ab-bb1d-7bd5aaf9163d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-54bd486f48-6dv4g" Oct 30 18:24:12.196969 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:12.196942 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30592a4b-d20f-44ab-bb1d-7bd5aaf9163d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-77db82-54bd486f48-6dv4g\" (UID: \"30592a4b-d20f-44ab-bb1d-7bd5aaf9163d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-54bd486f48-6dv4g" Oct 30 18:24:12.221066 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:12.220893 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6lkt\" (UniqueName: \"kubernetes.io/projected/30592a4b-d20f-44ab-bb1d-7bd5aaf9163d-kube-api-access-j6lkt\") pod \"payload-processing-e2e-ipp-a-77db82-54bd486f48-6dv4g\" (UID: \"30592a4b-d20f-44ab-bb1d-7bd5aaf9163d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-54bd486f48-6dv4g" Oct 30 18:24:12.256475 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:12.255088 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-54bd486f48-6dv4g" Oct 30 18:24:12.292671 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:12.292622 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-1fae38-d874c668-m42lh" Oct 30 18:24:12.487854 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:12.487818 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-546655d4c5-knrgr"] Oct 30 18:24:12.488889 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:24:12.488858 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f5acd91_9868_400b_b97f_f5416477f590.slice/crio-0e81072bd7ae57ec1b69769b2feef53c3087e120c89001253c819398c9e3b01e WatchSource:0}: Error finding container 0e81072bd7ae57ec1b69769b2feef53c3087e120c89001253c819398c9e3b01e: Status 404 returned error can't find the container with id 0e81072bd7ae57ec1b69769b2feef53c3087e120c89001253c819398c9e3b01e Oct 30 18:24:12.604183 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:24:12.604154 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30592a4b_d20f_44ab_bb1d_7bd5aaf9163d.slice/crio-71840eef602bbd78ea0b016f7d9ce742fc894857af727c4d4656928b9e8f090e WatchSource:0}: Error finding container 71840eef602bbd78ea0b016f7d9ce742fc894857af727c4d4656928b9e8f090e: Status 404 returned error can't find the container with id 71840eef602bbd78ea0b016f7d9ce742fc894857af727c4d4656928b9e8f090e Oct 30 18:24:12.608720 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:12.608693 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-77db82-54bd486f48-6dv4g"] Oct 30 18:24:13.491553 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:13.491501 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-54bd486f48-6dv4g" event={"ID":"30592a4b-d20f-44ab-bb1d-7bd5aaf9163d","Type":"ContainerStarted","Data":"6d77307b5a4a24e88ac78ae2fed5f4c15842a29282e5c9032fed5fa7179ea4ff"} Oct 30 18:24:13.492110 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:13.491571 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-54bd486f48-6dv4g" event={"ID":"30592a4b-d20f-44ab-bb1d-7bd5aaf9163d","Type":"ContainerStarted","Data":"71840eef602bbd78ea0b016f7d9ce742fc894857af727c4d4656928b9e8f090e"} Oct 30 18:24:13.492110 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:13.491601 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-54bd486f48-6dv4g" Oct 30 18:24:13.493542 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:13.493506 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-546655d4c5-knrgr" event={"ID":"2f5acd91-9868-400b-b97f-f5416477f590","Type":"ContainerStarted","Data":"e06b4ba4ff79e5475bf94f36428dbef2a4109adb548f18d16a150ca91556bb13"} Oct 30 18:24:13.493684 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:13.493551 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-546655d4c5-knrgr" event={"ID":"2f5acd91-9868-400b-b97f-f5416477f590","Type":"ContainerStarted","Data":"0e81072bd7ae57ec1b69769b2feef53c3087e120c89001253c819398c9e3b01e"} Oct 30 18:24:13.493684 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:13.493575 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-546655d4c5-knrgr" Oct 30 18:24:13.537538 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:13.537478 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-54bd486f48-6dv4g" podStartSLOduration=2.537461345 podStartE2EDuration="2.537461345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:13.536423207 +0000 UTC m=+1906.024060241" watchObservedRunningTime="2026-10-30 18:24:13.537461345 +0000 UTC m=+1906.025098379" Oct 30 18:24:13.588101 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:13.588047 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-546655d4c5-knrgr" podStartSLOduration=2.588031927 podStartE2EDuration="2.588031927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:13.585162266 +0000 UTC m=+1906.072799302" watchObservedRunningTime="2026-10-30 18:24:13.588031927 +0000 UTC m=+1906.075668961" Oct 30 18:24:14.274173 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:14.274134 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q"] Oct 30 18:24:14.279916 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:14.279891 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q" Oct 30 18:24:14.303478 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:14.303219 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-1fae38-serving-cert\"" Oct 30 18:24:14.305033 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:14.305006 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q"] Oct 30 18:24:14.346343 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:14.346295 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8ct7\" (UniqueName: \"kubernetes.io/projected/13d35525-5ffe-467e-a701-3200ca4facfd-kube-api-access-m8ct7\") pod \"maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q\" (UID: \"13d35525-5ffe-467e-a701-3200ca4facfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q" Oct 30 18:24:14.346548 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:14.346414 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d35525-5ffe-467e-a701-3200ca4facfd-metrics-certs\") pod \"maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q\" (UID: \"13d35525-5ffe-467e-a701-3200ca4facfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q" Oct 30 18:24:14.346548 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:14.346510 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d35525-5ffe-467e-a701-3200ca4facfd-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q\" (UID: \"13d35525-5ffe-467e-a701-3200ca4facfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q" Oct 30 18:24:14.346548 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:14.346533 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d35525-5ffe-467e-a701-3200ca4facfd-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q\" (UID: \"13d35525-5ffe-467e-a701-3200ca4facfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q" Oct 30 18:24:14.455347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:14.449888 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8ct7\" (UniqueName: \"kubernetes.io/projected/13d35525-5ffe-467e-a701-3200ca4facfd-kube-api-access-m8ct7\") pod \"maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q\" (UID: \"13d35525-5ffe-467e-a701-3200ca4facfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q" Oct 30 18:24:14.455347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:14.449987 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d35525-5ffe-467e-a701-3200ca4facfd-metrics-certs\") pod \"maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q\" (UID: \"13d35525-5ffe-467e-a701-3200ca4facfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q" Oct 30 18:24:14.455347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:14.450095 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d35525-5ffe-467e-a701-3200ca4facfd-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q\" (UID: \"13d35525-5ffe-467e-a701-3200ca4facfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q" Oct 30 18:24:14.455347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:14.450138 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d35525-5ffe-467e-a701-3200ca4facfd-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q\" (UID: \"13d35525-5ffe-467e-a701-3200ca4facfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q" Oct 30 18:24:14.455347 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:24:14.450455 4455 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-1fae38-serving-cert: secret "maas-api-e2e-praxis-a-1fae38-serving-cert" not found Oct 30 18:24:14.455347 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:24:14.450527 4455 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d35525-5ffe-467e-a701-3200ca4facfd-maas-api-tls podName:13d35525-5ffe-467e-a701-3200ca4facfd nodeName:}" failed. No retries permitted until 2026-10-30 18:24:14.950505585 +0000 UTC m=+1907.438142616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/13d35525-5ffe-467e-a701-3200ca4facfd-maas-api-tls") pod "maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q" (UID: "13d35525-5ffe-467e-a701-3200ca4facfd") : secret "maas-api-e2e-praxis-a-1fae38-serving-cert" not found Oct 30 18:24:14.480350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:14.472374 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d35525-5ffe-467e-a701-3200ca4facfd-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q\" (UID: \"13d35525-5ffe-467e-a701-3200ca4facfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q" Oct 30 18:24:14.487407 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:14.481783 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69" podUID="80b01b7a-2c9f-41b5-ba92-9ade95a89cc8" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.166:8443/health\": dial tcp 10.134.0.166:8443: connect: connection refused" Oct 30 18:24:14.521949 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:14.515981 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d35525-5ffe-467e-a701-3200ca4facfd-metrics-certs\") pod \"maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q\" (UID: \"13d35525-5ffe-467e-a701-3200ca4facfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q" Oct 30 18:24:14.548598 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:14.545378 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8ct7\" (UniqueName: \"kubernetes.io/projected/13d35525-5ffe-467e-a701-3200ca4facfd-kube-api-access-m8ct7\") pod \"maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q\" (UID: \"13d35525-5ffe-467e-a701-3200ca4facfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q" Oct 30 18:24:14.962016 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:14.961930 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d35525-5ffe-467e-a701-3200ca4facfd-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q\" (UID: \"13d35525-5ffe-467e-a701-3200ca4facfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q" Oct 30 18:24:14.964832 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:14.964806 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d35525-5ffe-467e-a701-3200ca4facfd-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q\" (UID: \"13d35525-5ffe-467e-a701-3200ca4facfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q" Oct 30 18:24:15.197742 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:15.197699 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q" Oct 30 18:24:15.593839 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:24:15.593804 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d35525_5ffe_467e_a701_3200ca4facfd.slice/crio-1eae9bd612d315f0bd50925b8ea58e3be0da8656ed07f9fce0f81d8ffc2ba4cd WatchSource:0}: Error finding container 1eae9bd612d315f0bd50925b8ea58e3be0da8656ed07f9fce0f81d8ffc2ba4cd: Status 404 returned error can't find the container with id 1eae9bd612d315f0bd50925b8ea58e3be0da8656ed07f9fce0f81d8ffc2ba4cd Oct 30 18:24:15.595430 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:15.595392 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q"] Oct 30 18:24:16.564119 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:16.564084 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q" event={"ID":"13d35525-5ffe-467e-a701-3200ca4facfd","Type":"ContainerStarted","Data":"bd27d377b75f2037c54c0198f100967d5da6a89954de07f9481871bf800926cf"} Oct 30 18:24:16.564373 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:16.564144 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q" event={"ID":"13d35525-5ffe-467e-a701-3200ca4facfd","Type":"ContainerStarted","Data":"1eae9bd612d315f0bd50925b8ea58e3be0da8656ed07f9fce0f81d8ffc2ba4cd"} Oct 30 18:24:16.564373 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:16.564195 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q" Oct 30 18:24:16.617510 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:16.617396 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q" podStartSLOduration=2.3100051329999998 podStartE2EDuration="2.617377765s" totalImagesPullingTime="307.372632ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:16.611134667 +0000 UTC m=+1909.098771703" watchObservedRunningTime="2026-10-30 18:24:16.617377765 +0000 UTC m=+1909.105014800" Oct 30 18:24:18.586371 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:18.586258 4455 generic.go:356] "Generic (PLEG): container finished" podID="80b01b7a-2c9f-41b5-ba92-9ade95a89cc8" containerID="8af70fe921a366f858d2bfde039bf60ee64e59652dee0cafdee698b46e80551f" exitCode=1 Oct 30 18:24:18.586371 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:18.586342 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69" event={"ID":"80b01b7a-2c9f-41b5-ba92-9ade95a89cc8","Type":"ContainerDied","Data":"8af70fe921a366f858d2bfde039bf60ee64e59652dee0cafdee698b46e80551f"} Oct 30 18:24:18.586893 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:18.586864 4455 scope.go:122] "RemoveContainer" containerID="8af70fe921a366f858d2bfde039bf60ee64e59652dee0cafdee698b46e80551f" Oct 30 18:24:19.603114 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:19.602935 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69" event={"ID":"80b01b7a-2c9f-41b5-ba92-9ade95a89cc8","Type":"ContainerStarted","Data":"c18a87dab12929ae8d261322054c87f35bc7585013af55fc369120b39423d8c4"} Oct 30 18:24:19.603762 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:19.603698 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69" Oct 30 18:24:22.582923 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:22.582886 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q" Oct 30 18:24:24.543031 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:24.542998 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-546655d4c5-knrgr" Oct 30 18:24:24.543469 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:24.543151 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-54bd486f48-6dv4g" Oct 30 18:24:25.618845 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:25.618815 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69" Oct 30 18:24:29.984778 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:29.984740 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7"] Oct 30 18:24:29.996391 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:29.994919 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" Oct 30 18:24:30.025050 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:30.025021 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-77db82-serving-cert\"" Oct 30 18:24:30.032676 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:30.032646 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7"] Oct 30 18:24:30.057888 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:30.057848 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e30a2f8-aa4d-4dea-8468-aa09e6100c58-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7\" (UID: \"5e30a2f8-aa4d-4dea-8468-aa09e6100c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" Oct 30 18:24:30.058056 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:30.057992 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e30a2f8-aa4d-4dea-8468-aa09e6100c58-maas-api-tls\") pod \"maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7\" (UID: \"5e30a2f8-aa4d-4dea-8468-aa09e6100c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" Oct 30 18:24:30.058271 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:30.058251 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e30a2f8-aa4d-4dea-8468-aa09e6100c58-metrics-certs\") pod \"maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7\" (UID: \"5e30a2f8-aa4d-4dea-8468-aa09e6100c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" Oct 30 18:24:30.058389 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:30.058302 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz2b9\" (UniqueName: \"kubernetes.io/projected/5e30a2f8-aa4d-4dea-8468-aa09e6100c58-kube-api-access-cz2b9\") pod \"maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7\" (UID: \"5e30a2f8-aa4d-4dea-8468-aa09e6100c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" Oct 30 18:24:30.159872 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:30.159771 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e30a2f8-aa4d-4dea-8468-aa09e6100c58-metrics-certs\") pod \"maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7\" (UID: \"5e30a2f8-aa4d-4dea-8468-aa09e6100c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" Oct 30 18:24:30.159872 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:30.159839 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz2b9\" (UniqueName: \"kubernetes.io/projected/5e30a2f8-aa4d-4dea-8468-aa09e6100c58-kube-api-access-cz2b9\") pod \"maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7\" (UID: \"5e30a2f8-aa4d-4dea-8468-aa09e6100c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" Oct 30 18:24:30.159872 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:30.159873 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e30a2f8-aa4d-4dea-8468-aa09e6100c58-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7\" (UID: \"5e30a2f8-aa4d-4dea-8468-aa09e6100c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" Oct 30 18:24:30.160176 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:30.160085 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e30a2f8-aa4d-4dea-8468-aa09e6100c58-maas-api-tls\") pod \"maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7\" (UID: \"5e30a2f8-aa4d-4dea-8468-aa09e6100c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" Oct 30 18:24:30.160261 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:24:30.160239 4455 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-serving-cert: secret "maas-api-e2e-ipp-a-77db82-serving-cert" not found Oct 30 18:24:30.160311 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:24:30.160304 4455 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e30a2f8-aa4d-4dea-8468-aa09e6100c58-maas-api-tls podName:5e30a2f8-aa4d-4dea-8468-aa09e6100c58 nodeName:}" failed. No retries permitted until 2026-10-30 18:24:30.660286843 +0000 UTC m=+1923.147923855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5e30a2f8-aa4d-4dea-8468-aa09e6100c58-maas-api-tls") pod "maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" (UID: "5e30a2f8-aa4d-4dea-8468-aa09e6100c58") : secret "maas-api-e2e-ipp-a-77db82-serving-cert" not found Oct 30 18:24:30.160646 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:30.160583 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e30a2f8-aa4d-4dea-8468-aa09e6100c58-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7\" (UID: \"5e30a2f8-aa4d-4dea-8468-aa09e6100c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" Oct 30 18:24:30.163031 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:30.163004 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e30a2f8-aa4d-4dea-8468-aa09e6100c58-metrics-certs\") pod \"maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7\" (UID: \"5e30a2f8-aa4d-4dea-8468-aa09e6100c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" Oct 30 18:24:30.185716 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:30.185681 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz2b9\" (UniqueName: \"kubernetes.io/projected/5e30a2f8-aa4d-4dea-8468-aa09e6100c58-kube-api-access-cz2b9\") pod \"maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7\" (UID: \"5e30a2f8-aa4d-4dea-8468-aa09e6100c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" Oct 30 18:24:30.405419 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:24:30.405352 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:24:30.670357 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:30.670308 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e30a2f8-aa4d-4dea-8468-aa09e6100c58-maas-api-tls\") pod \"maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7\" (UID: \"5e30a2f8-aa4d-4dea-8468-aa09e6100c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" Oct 30 18:24:30.673122 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:30.673091 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e30a2f8-aa4d-4dea-8468-aa09e6100c58-maas-api-tls\") pod \"maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7\" (UID: \"5e30a2f8-aa4d-4dea-8468-aa09e6100c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" Oct 30 18:24:30.921398 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:30.917450 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" Oct 30 18:24:33.552307 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:24:33.552268 4455 log.go:32] "RunPodSandbox from runtime service failed" err=< Oct 30 18:24:33.552307 ip-10-0-130-84 kubenswrapper[4455]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7_odh-ai-gateway-infra_5e30a2f8-aa4d-4dea-8468-aa09e6100c58_0(17fb602f2287b1351f1c62208f539ee7ae1604306f9d8a2b8944ae7414dc794e): error adding pod odh-ai-gateway-infra_maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"17fb602f2287b1351f1c62208f539ee7ae1604306f9d8a2b8944ae7414dc794e" Netns:"/var/run/netns/d6badb47-10cf-4722-98f8-ec25898a55bb" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7;K8S_POD_INFRA_CONTAINER_ID=17fb602f2287b1351f1c62208f539ee7ae1604306f9d8a2b8944ae7414dc794e;K8S_POD_UID=5e30a2f8-aa4d-4dea-8468-aa09e6100c58" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7/5e30a2f8-aa4d-4dea-8468-aa09e6100c58]: error waiting for pod: Get "https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7?timeout=1m0s": context deadline exceeded Oct 30 18:24:33.552307 ip-10-0-130-84 kubenswrapper[4455]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 18:24:33.552307 ip-10-0-130-84 kubenswrapper[4455]: > Oct 30 18:24:33.552795 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:24:33.552370 4455 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Oct 30 18:24:33.552795 ip-10-0-130-84 kubenswrapper[4455]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7_odh-ai-gateway-infra_5e30a2f8-aa4d-4dea-8468-aa09e6100c58_0(17fb602f2287b1351f1c62208f539ee7ae1604306f9d8a2b8944ae7414dc794e): error adding pod odh-ai-gateway-infra_maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"17fb602f2287b1351f1c62208f539ee7ae1604306f9d8a2b8944ae7414dc794e" Netns:"/var/run/netns/d6badb47-10cf-4722-98f8-ec25898a55bb" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7;K8S_POD_INFRA_CONTAINER_ID=17fb602f2287b1351f1c62208f539ee7ae1604306f9d8a2b8944ae7414dc794e;K8S_POD_UID=5e30a2f8-aa4d-4dea-8468-aa09e6100c58" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7/5e30a2f8-aa4d-4dea-8468-aa09e6100c58]: error waiting for pod: Get "https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7?timeout=1m0s": context deadline exceeded Oct 30 18:24:33.552795 ip-10-0-130-84 kubenswrapper[4455]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 18:24:33.552795 ip-10-0-130-84 kubenswrapper[4455]: > pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" Oct 30 18:24:33.552795 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:24:33.552396 4455 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=< Oct 30 18:24:33.552795 ip-10-0-130-84 kubenswrapper[4455]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7_odh-ai-gateway-infra_5e30a2f8-aa4d-4dea-8468-aa09e6100c58_0(17fb602f2287b1351f1c62208f539ee7ae1604306f9d8a2b8944ae7414dc794e): error adding pod odh-ai-gateway-infra_maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"17fb602f2287b1351f1c62208f539ee7ae1604306f9d8a2b8944ae7414dc794e" Netns:"/var/run/netns/d6badb47-10cf-4722-98f8-ec25898a55bb" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7;K8S_POD_INFRA_CONTAINER_ID=17fb602f2287b1351f1c62208f539ee7ae1604306f9d8a2b8944ae7414dc794e;K8S_POD_UID=5e30a2f8-aa4d-4dea-8468-aa09e6100c58" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7/5e30a2f8-aa4d-4dea-8468-aa09e6100c58]: error waiting for pod: Get "https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7?timeout=1m0s": context deadline exceeded Oct 30 18:24:33.552795 ip-10-0-130-84 kubenswrapper[4455]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 18:24:33.552795 ip-10-0-130-84 kubenswrapper[4455]: > pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" Oct 30 18:24:33.552795 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:24:33.552492 4455 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7_odh-ai-gateway-infra(5e30a2f8-aa4d-4dea-8468-aa09e6100c58)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7_odh-ai-gateway-infra(5e30a2f8-aa4d-4dea-8468-aa09e6100c58)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7_odh-ai-gateway-infra_5e30a2f8-aa4d-4dea-8468-aa09e6100c58_0(17fb602f2287b1351f1c62208f539ee7ae1604306f9d8a2b8944ae7414dc794e): error adding pod odh-ai-gateway-infra_maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"17fb602f2287b1351f1c62208f539ee7ae1604306f9d8a2b8944ae7414dc794e\\\" Netns:\\\"/var/run/netns/d6badb47-10cf-4722-98f8-ec25898a55bb\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7;K8S_POD_INFRA_CONTAINER_ID=17fb602f2287b1351f1c62208f539ee7ae1604306f9d8a2b8944ae7414dc794e;K8S_POD_UID=5e30a2f8-aa4d-4dea-8468-aa09e6100c58\\\" Path:\\\"\\\" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7/5e30a2f8-aa4d-4dea-8468-aa09e6100c58]: error waiting for pod: Get \\\"https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" podUID="5e30a2f8-aa4d-4dea-8468-aa09e6100c58" Oct 30 18:24:33.747187 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:33.747155 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" Oct 30 18:24:33.747539 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:33.747521 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" Oct 30 18:24:36.357919 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:24:36.357880 4455 log.go:32] "RunPodSandbox from runtime service failed" err=< Oct 30 18:24:36.357919 ip-10-0-130-84 kubenswrapper[4455]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7_odh-ai-gateway-infra_5e30a2f8-aa4d-4dea-8468-aa09e6100c58_0(a9342941f148c4c5197b45c1398d997884cdb6efe7f9a41073bd91bb60d9e47c): error adding pod odh-ai-gateway-infra_maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"a9342941f148c4c5197b45c1398d997884cdb6efe7f9a41073bd91bb60d9e47c" Netns:"/var/run/netns/53e80cb8-2ada-4269-badf-d9e40d30b835" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7;K8S_POD_INFRA_CONTAINER_ID=a9342941f148c4c5197b45c1398d997884cdb6efe7f9a41073bd91bb60d9e47c;K8S_POD_UID=5e30a2f8-aa4d-4dea-8468-aa09e6100c58" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7/5e30a2f8-aa4d-4dea-8468-aa09e6100c58]: error waiting for pod: Get "https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7?timeout=1m0s": context deadline exceeded Oct 30 18:24:36.357919 ip-10-0-130-84 kubenswrapper[4455]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 18:24:36.357919 ip-10-0-130-84 kubenswrapper[4455]: > Oct 30 18:24:36.358547 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:24:36.357935 4455 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Oct 30 18:24:36.358547 ip-10-0-130-84 kubenswrapper[4455]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7_odh-ai-gateway-infra_5e30a2f8-aa4d-4dea-8468-aa09e6100c58_0(a9342941f148c4c5197b45c1398d997884cdb6efe7f9a41073bd91bb60d9e47c): error adding pod odh-ai-gateway-infra_maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"a9342941f148c4c5197b45c1398d997884cdb6efe7f9a41073bd91bb60d9e47c" Netns:"/var/run/netns/53e80cb8-2ada-4269-badf-d9e40d30b835" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7;K8S_POD_INFRA_CONTAINER_ID=a9342941f148c4c5197b45c1398d997884cdb6efe7f9a41073bd91bb60d9e47c;K8S_POD_UID=5e30a2f8-aa4d-4dea-8468-aa09e6100c58" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7/5e30a2f8-aa4d-4dea-8468-aa09e6100c58]: error waiting for pod: Get "https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7?timeout=1m0s": context deadline exceeded Oct 30 18:24:36.358547 ip-10-0-130-84 kubenswrapper[4455]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 18:24:36.358547 ip-10-0-130-84 kubenswrapper[4455]: > pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" Oct 30 18:24:36.358547 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:24:36.357952 4455 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=< Oct 30 18:24:36.358547 ip-10-0-130-84 kubenswrapper[4455]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7_odh-ai-gateway-infra_5e30a2f8-aa4d-4dea-8468-aa09e6100c58_0(a9342941f148c4c5197b45c1398d997884cdb6efe7f9a41073bd91bb60d9e47c): error adding pod odh-ai-gateway-infra_maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"a9342941f148c4c5197b45c1398d997884cdb6efe7f9a41073bd91bb60d9e47c" Netns:"/var/run/netns/53e80cb8-2ada-4269-badf-d9e40d30b835" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7;K8S_POD_INFRA_CONTAINER_ID=a9342941f148c4c5197b45c1398d997884cdb6efe7f9a41073bd91bb60d9e47c;K8S_POD_UID=5e30a2f8-aa4d-4dea-8468-aa09e6100c58" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7/5e30a2f8-aa4d-4dea-8468-aa09e6100c58]: error waiting for pod: Get "https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7?timeout=1m0s": context deadline exceeded Oct 30 18:24:36.358547 ip-10-0-130-84 kubenswrapper[4455]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 18:24:36.358547 ip-10-0-130-84 kubenswrapper[4455]: > pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" Oct 30 18:24:36.358547 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:24:36.358020 4455 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7_odh-ai-gateway-infra(5e30a2f8-aa4d-4dea-8468-aa09e6100c58)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7_odh-ai-gateway-infra(5e30a2f8-aa4d-4dea-8468-aa09e6100c58)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7_odh-ai-gateway-infra_5e30a2f8-aa4d-4dea-8468-aa09e6100c58_0(a9342941f148c4c5197b45c1398d997884cdb6efe7f9a41073bd91bb60d9e47c): error adding pod odh-ai-gateway-infra_maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"a9342941f148c4c5197b45c1398d997884cdb6efe7f9a41073bd91bb60d9e47c\\\" Netns:\\\"/var/run/netns/53e80cb8-2ada-4269-badf-d9e40d30b835\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7;K8S_POD_INFRA_CONTAINER_ID=a9342941f148c4c5197b45c1398d997884cdb6efe7f9a41073bd91bb60d9e47c;K8S_POD_UID=5e30a2f8-aa4d-4dea-8468-aa09e6100c58\\\" Path:\\\"\\\" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7/5e30a2f8-aa4d-4dea-8468-aa09e6100c58]: error waiting for pod: Get \\\"https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" podUID="5e30a2f8-aa4d-4dea-8468-aa09e6100c58" Oct 30 18:24:36.846027 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:36.845991 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ef208a-6c5cd4b4d-wrtmz"] Oct 30 18:24:36.882126 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:36.882090 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ef208a-9ccb8d8f5-jqhpw"] Oct 30 18:24:41.846619 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:41.846523 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-ef208a-6c5cd4b4d-wrtmz" podUID="58f7f060-dc48-4a8d-87a1-a760d0351fb6" containerName="payload-processing" containerID="cri-o://c31606582efa2941bf89b88cd39ad46df02553998e98883f0e73acbbadd60568" gracePeriod=25 Oct 30 18:24:41.882399 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:41.882314 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ef208a-9ccb8d8f5-jqhpw" podUID="0ad56482-fee8-47da-85c7-b870fa1e1aeb" containerName="payload-pre-processing" containerID="cri-o://911b93172fae9dcc7a6bfdd099914e3a09d8197df2c59e936cfc451ec1e10855" gracePeriod=25 Oct 30 18:24:42.196931 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:42.196541 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ef208a-6c5cd4b4d-wrtmz" Oct 30 18:24:42.244529 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:42.244500 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ef208a-9ccb8d8f5-jqhpw" Oct 30 18:24:42.330573 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:42.330537 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ad56482-fee8-47da-85c7-b870fa1e1aeb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad56482-fee8-47da-85c7-b870fa1e1aeb-plugins-config-volume\") pod \"0ad56482-fee8-47da-85c7-b870fa1e1aeb\" (UID: \"0ad56482-fee8-47da-85c7-b870fa1e1aeb\") " Oct 30 18:24:42.330772 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:42.330693 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58f7f060-dc48-4a8d-87a1-a760d0351fb6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f7f060-dc48-4a8d-87a1-a760d0351fb6-plugins-config-volume\") pod \"58f7f060-dc48-4a8d-87a1-a760d0351fb6\" (UID: \"58f7f060-dc48-4a8d-87a1-a760d0351fb6\") " Oct 30 18:24:42.330772 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:42.330738 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ad56482-fee8-47da-85c7-b870fa1e1aeb-kube-api-access-wff5v\" (UniqueName: \"kubernetes.io/projected/0ad56482-fee8-47da-85c7-b870fa1e1aeb-kube-api-access-wff5v\") pod \"0ad56482-fee8-47da-85c7-b870fa1e1aeb\" (UID: \"0ad56482-fee8-47da-85c7-b870fa1e1aeb\") " Oct 30 18:24:42.330952 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:42.330915 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58f7f060-dc48-4a8d-87a1-a760d0351fb6-kube-api-access-7799m\" (UniqueName: \"kubernetes.io/projected/58f7f060-dc48-4a8d-87a1-a760d0351fb6-kube-api-access-7799m\") pod \"58f7f060-dc48-4a8d-87a1-a760d0351fb6\" (UID: \"58f7f060-dc48-4a8d-87a1-a760d0351fb6\") " Oct 30 18:24:42.331037 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:42.331007 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ad56482-fee8-47da-85c7-b870fa1e1aeb-plugins-config-volume" pod "0ad56482-fee8-47da-85c7-b870fa1e1aeb" (UID: "0ad56482-fee8-47da-85c7-b870fa1e1aeb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:42.331410 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:42.331117 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58f7f060-dc48-4a8d-87a1-a760d0351fb6-plugins-config-volume" pod "58f7f060-dc48-4a8d-87a1-a760d0351fb6" (UID: "58f7f060-dc48-4a8d-87a1-a760d0351fb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:42.331624 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:42.331605 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad56482-fee8-47da-85c7-b870fa1e1aeb-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:24:42.331624 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:42.331626 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f7f060-dc48-4a8d-87a1-a760d0351fb6-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:24:42.333393 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:42.333359 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ad56482-fee8-47da-85c7-b870fa1e1aeb-kube-api-access-wff5v" pod "0ad56482-fee8-47da-85c7-b870fa1e1aeb" (UID: "0ad56482-fee8-47da-85c7-b870fa1e1aeb"). InnerVolumeSpecName "kube-api-access-wff5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:42.333741 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:42.333715 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58f7f060-dc48-4a8d-87a1-a760d0351fb6-kube-api-access-7799m" pod "58f7f060-dc48-4a8d-87a1-a760d0351fb6" (UID: "58f7f060-dc48-4a8d-87a1-a760d0351fb6"). InnerVolumeSpecName "kube-api-access-7799m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:42.433755 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:42.433667 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wff5v\" (UniqueName: \"kubernetes.io/projected/0ad56482-fee8-47da-85c7-b870fa1e1aeb-kube-api-access-wff5v\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:24:42.433755 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:42.433700 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7799m\" (UniqueName: \"kubernetes.io/projected/58f7f060-dc48-4a8d-87a1-a760d0351fb6-kube-api-access-7799m\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:24:42.904880 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:42.904825 4455 generic.go:356] "Generic (PLEG): container finished" podID="58f7f060-dc48-4a8d-87a1-a760d0351fb6" containerID="c31606582efa2941bf89b88cd39ad46df02553998e98883f0e73acbbadd60568" exitCode=0 Oct 30 18:24:42.905563 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:42.905536 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ef208a-6c5cd4b4d-wrtmz" event={"ID":"58f7f060-dc48-4a8d-87a1-a760d0351fb6","Type":"ContainerDied","Data":"c31606582efa2941bf89b88cd39ad46df02553998e98883f0e73acbbadd60568"} Oct 30 18:24:42.905839 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:42.905819 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ef208a-6c5cd4b4d-wrtmz" event={"ID":"58f7f060-dc48-4a8d-87a1-a760d0351fb6","Type":"ContainerDied","Data":"1b859ef208a9a647b5f028757ea739d954d8489a5784bf4f90115e7ff30d783c"} Oct 30 18:24:42.905987 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:42.905974 4455 scope.go:122] "RemoveContainer" containerID="c31606582efa2941bf89b88cd39ad46df02553998e98883f0e73acbbadd60568" Oct 30 18:24:42.906254 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:42.906237 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ef208a-6c5cd4b4d-wrtmz" Oct 30 18:24:42.998367 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:42.995438 4455 generic.go:356] "Generic (PLEG): container finished" podID="0ad56482-fee8-47da-85c7-b870fa1e1aeb" containerID="911b93172fae9dcc7a6bfdd099914e3a09d8197df2c59e936cfc451ec1e10855" exitCode=0 Oct 30 18:24:42.998367 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:42.995556 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ef208a-9ccb8d8f5-jqhpw" event={"ID":"0ad56482-fee8-47da-85c7-b870fa1e1aeb","Type":"ContainerDied","Data":"911b93172fae9dcc7a6bfdd099914e3a09d8197df2c59e936cfc451ec1e10855"} Oct 30 18:24:42.998367 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:42.995577 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ef208a-9ccb8d8f5-jqhpw" Oct 30 18:24:42.998367 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:42.995655 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ef208a-9ccb8d8f5-jqhpw" event={"ID":"0ad56482-fee8-47da-85c7-b870fa1e1aeb","Type":"ContainerDied","Data":"e3027dde22628524a991977fd7e4ae18fb30a2349127f2f360cb26f06b85f795"} Oct 30 18:24:43.042708 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:43.041773 4455 scope.go:122] "RemoveContainer" containerID="c31606582efa2941bf89b88cd39ad46df02553998e98883f0e73acbbadd60568" Oct 30 18:24:43.042708 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:24:43.042145 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c31606582efa2941bf89b88cd39ad46df02553998e98883f0e73acbbadd60568\": container with ID starting with c31606582efa2941bf89b88cd39ad46df02553998e98883f0e73acbbadd60568 not found: ID does not exist" containerID="c31606582efa2941bf89b88cd39ad46df02553998e98883f0e73acbbadd60568" Oct 30 18:24:43.042708 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:43.042173 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c31606582efa2941bf89b88cd39ad46df02553998e98883f0e73acbbadd60568"} err="failed to get container status \"c31606582efa2941bf89b88cd39ad46df02553998e98883f0e73acbbadd60568\": rpc error: code = NotFound desc = could not find container \"c31606582efa2941bf89b88cd39ad46df02553998e98883f0e73acbbadd60568\": container with ID starting with c31606582efa2941bf89b88cd39ad46df02553998e98883f0e73acbbadd60568 not found: ID does not exist" Oct 30 18:24:43.042708 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:43.042196 4455 scope.go:122] "RemoveContainer" containerID="911b93172fae9dcc7a6bfdd099914e3a09d8197df2c59e936cfc451ec1e10855" Oct 30 18:24:43.043250 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:43.043206 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ef208a-6c5cd4b4d-wrtmz"] Oct 30 18:24:43.050789 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:43.050761 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ef208a-6c5cd4b4d-wrtmz"] Oct 30 18:24:43.055915 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:43.055893 4455 scope.go:122] "RemoveContainer" containerID="911b93172fae9dcc7a6bfdd099914e3a09d8197df2c59e936cfc451ec1e10855" Oct 30 18:24:43.056224 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:24:43.056203 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"911b93172fae9dcc7a6bfdd099914e3a09d8197df2c59e936cfc451ec1e10855\": container with ID starting with 911b93172fae9dcc7a6bfdd099914e3a09d8197df2c59e936cfc451ec1e10855 not found: ID does not exist" containerID="911b93172fae9dcc7a6bfdd099914e3a09d8197df2c59e936cfc451ec1e10855" Oct 30 18:24:43.056272 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:43.056234 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"911b93172fae9dcc7a6bfdd099914e3a09d8197df2c59e936cfc451ec1e10855"} err="failed to get container status \"911b93172fae9dcc7a6bfdd099914e3a09d8197df2c59e936cfc451ec1e10855\": rpc error: code = NotFound desc = could not find container \"911b93172fae9dcc7a6bfdd099914e3a09d8197df2c59e936cfc451ec1e10855\": container with ID starting with 911b93172fae9dcc7a6bfdd099914e3a09d8197df2c59e936cfc451ec1e10855 not found: ID does not exist" Oct 30 18:24:43.104332 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:43.104283 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ef208a-9ccb8d8f5-jqhpw"] Oct 30 18:24:43.109450 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:43.109422 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ef208a-9ccb8d8f5-jqhpw"] Oct 30 18:24:43.639873 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:43.639841 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ad56482-fee8-47da-85c7-b870fa1e1aeb" path="/var/lib/kubelet/pods/0ad56482-fee8-47da-85c7-b870fa1e1aeb/volumes" Oct 30 18:24:43.640379 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:43.640358 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58f7f060-dc48-4a8d-87a1-a760d0351fb6" path="/var/lib/kubelet/pods/58f7f060-dc48-4a8d-87a1-a760d0351fb6/volumes" Oct 30 18:24:45.824975 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:45.824941 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78"] Oct 30 18:24:45.825527 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:45.825173 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78" podUID="a1080e9c-6b0d-4aef-90a6-e041970b9a35" containerName="maas-api" containerID="cri-o://f61d29e9ecb27f3531335c5aae3f8ad9ef5aeb5f411c8f091d4d81b7db3a6047" gracePeriod=30 Oct 30 18:24:46.138955 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:46.138854 4455 generic.go:356] "Generic (PLEG): container finished" podID="a1080e9c-6b0d-4aef-90a6-e041970b9a35" containerID="f61d29e9ecb27f3531335c5aae3f8ad9ef5aeb5f411c8f091d4d81b7db3a6047" exitCode=0 Oct 30 18:24:46.154352 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:46.143366 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78" event={"ID":"a1080e9c-6b0d-4aef-90a6-e041970b9a35","Type":"ContainerDied","Data":"f61d29e9ecb27f3531335c5aae3f8ad9ef5aeb5f411c8f091d4d81b7db3a6047"} Oct 30 18:24:46.433489 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:46.430532 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78" Oct 30 18:24:46.569666 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:46.569620 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1080e9c-6b0d-4aef-90a6-e041970b9a35-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1080e9c-6b0d-4aef-90a6-e041970b9a35-maas-api-tls\") pod \"a1080e9c-6b0d-4aef-90a6-e041970b9a35\" (UID: \"a1080e9c-6b0d-4aef-90a6-e041970b9a35\") " Oct 30 18:24:46.569886 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:46.569710 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1080e9c-6b0d-4aef-90a6-e041970b9a35-kube-api-access-kkhf7\" (UniqueName: \"kubernetes.io/projected/a1080e9c-6b0d-4aef-90a6-e041970b9a35-kube-api-access-kkhf7\") pod \"a1080e9c-6b0d-4aef-90a6-e041970b9a35\" (UID: \"a1080e9c-6b0d-4aef-90a6-e041970b9a35\") " Oct 30 18:24:46.569886 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:46.569848 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1080e9c-6b0d-4aef-90a6-e041970b9a35-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1080e9c-6b0d-4aef-90a6-e041970b9a35-metrics-certs\") pod \"a1080e9c-6b0d-4aef-90a6-e041970b9a35\" (UID: \"a1080e9c-6b0d-4aef-90a6-e041970b9a35\") " Oct 30 18:24:46.570032 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:46.570014 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1080e9c-6b0d-4aef-90a6-e041970b9a35-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1080e9c-6b0d-4aef-90a6-e041970b9a35-openshift-service-ca\") pod \"a1080e9c-6b0d-4aef-90a6-e041970b9a35\" (UID: \"a1080e9c-6b0d-4aef-90a6-e041970b9a35\") " Oct 30 18:24:46.571250 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:46.571223 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1080e9c-6b0d-4aef-90a6-e041970b9a35-openshift-service-ca" pod "a1080e9c-6b0d-4aef-90a6-e041970b9a35" (UID: "a1080e9c-6b0d-4aef-90a6-e041970b9a35"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:46.572521 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:46.572488 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1080e9c-6b0d-4aef-90a6-e041970b9a35-maas-api-tls" pod "a1080e9c-6b0d-4aef-90a6-e041970b9a35" (UID: "a1080e9c-6b0d-4aef-90a6-e041970b9a35"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:24:46.572806 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:46.572779 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1080e9c-6b0d-4aef-90a6-e041970b9a35-kube-api-access-kkhf7" pod "a1080e9c-6b0d-4aef-90a6-e041970b9a35" (UID: "a1080e9c-6b0d-4aef-90a6-e041970b9a35"). InnerVolumeSpecName "kube-api-access-kkhf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:46.573251 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:46.573230 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1080e9c-6b0d-4aef-90a6-e041970b9a35-metrics-certs" pod "a1080e9c-6b0d-4aef-90a6-e041970b9a35" (UID: "a1080e9c-6b0d-4aef-90a6-e041970b9a35"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:24:46.672005 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:46.671970 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkhf7\" (UniqueName: \"kubernetes.io/projected/a1080e9c-6b0d-4aef-90a6-e041970b9a35-kube-api-access-kkhf7\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.672005 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:46.672002 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1080e9c-6b0d-4aef-90a6-e041970b9a35-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.672218 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:46.672015 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1080e9c-6b0d-4aef-90a6-e041970b9a35-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.672218 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:46.672027 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1080e9c-6b0d-4aef-90a6-e041970b9a35-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.698784 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:46.694930 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d42172-54484dd44c-nn9fs"] Oct 30 18:24:47.191733 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:47.191707 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78" Oct 30 18:24:47.191733 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:47.191723 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78" event={"ID":"a1080e9c-6b0d-4aef-90a6-e041970b9a35","Type":"ContainerDied","Data":"00ec3f1292fb64224f9fd893ed0450b1de7214d4091ed4d9eb7b6dc6f2ccf37c"} Oct 30 18:24:47.192262 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:47.191784 4455 scope.go:122] "RemoveContainer" containerID="f61d29e9ecb27f3531335c5aae3f8ad9ef5aeb5f411c8f091d4d81b7db3a6047" Oct 30 18:24:47.260942 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:47.260903 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78"] Oct 30 18:24:47.267524 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:47.267497 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ef208a-cdbcf79f7-hww78"] Oct 30 18:24:47.631395 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:47.630310 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" Oct 30 18:24:47.634111 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:47.632383 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" Oct 30 18:24:47.678589 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:47.676889 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1080e9c-6b0d-4aef-90a6-e041970b9a35" path="/var/lib/kubelet/pods/a1080e9c-6b0d-4aef-90a6-e041970b9a35/volumes" Oct 30 18:24:50.254179 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:50.254147 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45"] Oct 30 18:24:50.254974 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:50.254957 4455 cpu_manager.go:414] "containerMap: removing container" podUID="58f7f060-dc48-4a8d-87a1-a760d0351fb6" containerName="payload-processing" Oct 30 18:24:50.255039 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:50.254976 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58f7f060-dc48-4a8d-87a1-a760d0351fb6" containerName="payload-processing" Oct 30 18:24:50.255039 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:50.254993 4455 cpu_manager.go:414] "containerMap: removing container" podUID="a1080e9c-6b0d-4aef-90a6-e041970b9a35" containerName="maas-api" Oct 30 18:24:50.255039 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:50.254997 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1080e9c-6b0d-4aef-90a6-e041970b9a35" containerName="maas-api" Oct 30 18:24:50.255039 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:50.255016 4455 cpu_manager.go:414] "containerMap: removing container" podUID="0ad56482-fee8-47da-85c7-b870fa1e1aeb" containerName="payload-pre-processing" Oct 30 18:24:50.255039 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:50.255022 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ad56482-fee8-47da-85c7-b870fa1e1aeb" containerName="payload-pre-processing" Oct 30 18:24:50.255200 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:50.255145 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="58f7f060-dc48-4a8d-87a1-a760d0351fb6" containerName="payload-processing" Oct 30 18:24:50.255200 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:50.255155 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1080e9c-6b0d-4aef-90a6-e041970b9a35" containerName="maas-api" Oct 30 18:24:50.255200 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:50.255164 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ad56482-fee8-47da-85c7-b870fa1e1aeb" containerName="payload-pre-processing" Oct 30 18:24:50.258917 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:50.258899 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" Oct 30 18:24:50.296511 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:50.296478 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45"] Oct 30 18:24:50.313304 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:24:50.313052 4455 log.go:32] "RunPodSandbox from runtime service failed" err=< Oct 30 18:24:50.313304 ip-10-0-130-84 kubenswrapper[4455]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7_odh-ai-gateway-infra_5e30a2f8-aa4d-4dea-8468-aa09e6100c58_0(c753d12c68cb5de7f2a6734265a4dc96531ab3587b83eff699eb70f43c02a8c9): error adding pod odh-ai-gateway-infra_maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"c753d12c68cb5de7f2a6734265a4dc96531ab3587b83eff699eb70f43c02a8c9" Netns:"/var/run/netns/b5d22cdf-3111-4441-9b1f-66d7ed6a6c94" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7;K8S_POD_INFRA_CONTAINER_ID=c753d12c68cb5de7f2a6734265a4dc96531ab3587b83eff699eb70f43c02a8c9;K8S_POD_UID=5e30a2f8-aa4d-4dea-8468-aa09e6100c58" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7/5e30a2f8-aa4d-4dea-8468-aa09e6100c58]: error waiting for pod: Get "https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7?timeout=1m0s": context deadline exceeded Oct 30 18:24:50.313304 ip-10-0-130-84 kubenswrapper[4455]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 18:24:50.313304 ip-10-0-130-84 kubenswrapper[4455]: > Oct 30 18:24:50.313304 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:24:50.313111 4455 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Oct 30 18:24:50.313304 ip-10-0-130-84 kubenswrapper[4455]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7_odh-ai-gateway-infra_5e30a2f8-aa4d-4dea-8468-aa09e6100c58_0(c753d12c68cb5de7f2a6734265a4dc96531ab3587b83eff699eb70f43c02a8c9): error adding pod odh-ai-gateway-infra_maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"c753d12c68cb5de7f2a6734265a4dc96531ab3587b83eff699eb70f43c02a8c9" Netns:"/var/run/netns/b5d22cdf-3111-4441-9b1f-66d7ed6a6c94" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7;K8S_POD_INFRA_CONTAINER_ID=c753d12c68cb5de7f2a6734265a4dc96531ab3587b83eff699eb70f43c02a8c9;K8S_POD_UID=5e30a2f8-aa4d-4dea-8468-aa09e6100c58" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7/5e30a2f8-aa4d-4dea-8468-aa09e6100c58]: error waiting for pod: Get "https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7?timeout=1m0s": context deadline exceeded Oct 30 18:24:50.313304 ip-10-0-130-84 kubenswrapper[4455]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 18:24:50.313304 ip-10-0-130-84 kubenswrapper[4455]: > pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" Oct 30 18:24:50.313304 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:24:50.313132 4455 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=< Oct 30 18:24:50.313304 ip-10-0-130-84 kubenswrapper[4455]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7_odh-ai-gateway-infra_5e30a2f8-aa4d-4dea-8468-aa09e6100c58_0(c753d12c68cb5de7f2a6734265a4dc96531ab3587b83eff699eb70f43c02a8c9): error adding pod odh-ai-gateway-infra_maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"c753d12c68cb5de7f2a6734265a4dc96531ab3587b83eff699eb70f43c02a8c9" Netns:"/var/run/netns/b5d22cdf-3111-4441-9b1f-66d7ed6a6c94" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7;K8S_POD_INFRA_CONTAINER_ID=c753d12c68cb5de7f2a6734265a4dc96531ab3587b83eff699eb70f43c02a8c9;K8S_POD_UID=5e30a2f8-aa4d-4dea-8468-aa09e6100c58" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7/5e30a2f8-aa4d-4dea-8468-aa09e6100c58]: error waiting for pod: Get "https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7?timeout=1m0s": context deadline exceeded Oct 30 18:24:50.313304 ip-10-0-130-84 kubenswrapper[4455]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 18:24:50.313304 ip-10-0-130-84 kubenswrapper[4455]: > pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" Oct 30 18:24:50.313304 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:24:50.313225 4455 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7_odh-ai-gateway-infra(5e30a2f8-aa4d-4dea-8468-aa09e6100c58)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7_odh-ai-gateway-infra(5e30a2f8-aa4d-4dea-8468-aa09e6100c58)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7_odh-ai-gateway-infra_5e30a2f8-aa4d-4dea-8468-aa09e6100c58_0(c753d12c68cb5de7f2a6734265a4dc96531ab3587b83eff699eb70f43c02a8c9): error adding pod odh-ai-gateway-infra_maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"c753d12c68cb5de7f2a6734265a4dc96531ab3587b83eff699eb70f43c02a8c9\\\" Netns:\\\"/var/run/netns/b5d22cdf-3111-4441-9b1f-66d7ed6a6c94\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7;K8S_POD_INFRA_CONTAINER_ID=c753d12c68cb5de7f2a6734265a4dc96531ab3587b83eff699eb70f43c02a8c9;K8S_POD_UID=5e30a2f8-aa4d-4dea-8468-aa09e6100c58\\\" Path:\\\"\\\" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7/5e30a2f8-aa4d-4dea-8468-aa09e6100c58]: error waiting for pod: Get \\\"https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" podUID="5e30a2f8-aa4d-4dea-8468-aa09e6100c58" Oct 30 18:24:50.341286 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:50.341257 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfbae34d-b472-4dd7-b862-dbb457726886-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45\" (UID: \"bfbae34d-b472-4dd7-b862-dbb457726886\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" Oct 30 18:24:50.341461 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:50.341299 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfbae34d-b472-4dd7-b862-dbb457726886-metrics-certs\") pod \"maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45\" (UID: \"bfbae34d-b472-4dd7-b862-dbb457726886\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" Oct 30 18:24:50.341461 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:50.341368 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfbae34d-b472-4dd7-b862-dbb457726886-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45\" (UID: \"bfbae34d-b472-4dd7-b862-dbb457726886\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" Oct 30 18:24:50.341600 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:50.341557 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-c2b103-serving-cert\"" Oct 30 18:24:50.341600 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:50.341562 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4jrn\" (UniqueName: \"kubernetes.io/projected/bfbae34d-b472-4dd7-b862-dbb457726886-kube-api-access-l4jrn\") pod \"maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45\" (UID: \"bfbae34d-b472-4dd7-b862-dbb457726886\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" Oct 30 18:24:50.464350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:50.456429 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4jrn\" (UniqueName: \"kubernetes.io/projected/bfbae34d-b472-4dd7-b862-dbb457726886-kube-api-access-l4jrn\") pod \"maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45\" (UID: \"bfbae34d-b472-4dd7-b862-dbb457726886\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" Oct 30 18:24:50.464350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:50.456646 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfbae34d-b472-4dd7-b862-dbb457726886-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45\" (UID: \"bfbae34d-b472-4dd7-b862-dbb457726886\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" Oct 30 18:24:50.464350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:50.456688 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfbae34d-b472-4dd7-b862-dbb457726886-metrics-certs\") pod \"maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45\" (UID: \"bfbae34d-b472-4dd7-b862-dbb457726886\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" Oct 30 18:24:50.464350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:50.456801 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfbae34d-b472-4dd7-b862-dbb457726886-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45\" (UID: \"bfbae34d-b472-4dd7-b862-dbb457726886\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" Oct 30 18:24:50.464350 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:24:50.457035 4455 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-serving-cert: secret "maas-api-e2e-praxis-b-c2b103-serving-cert" not found Oct 30 18:24:50.464350 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:24:50.457104 4455 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfbae34d-b472-4dd7-b862-dbb457726886-maas-api-tls podName:bfbae34d-b472-4dd7-b862-dbb457726886 nodeName:}" failed. No retries permitted until 2026-10-30 18:24:50.957083284 +0000 UTC m=+1943.444720299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bfbae34d-b472-4dd7-b862-dbb457726886-maas-api-tls") pod "maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" (UID: "bfbae34d-b472-4dd7-b862-dbb457726886") : secret "maas-api-e2e-praxis-b-c2b103-serving-cert" not found Oct 30 18:24:50.464350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:50.458386 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfbae34d-b472-4dd7-b862-dbb457726886-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45\" (UID: \"bfbae34d-b472-4dd7-b862-dbb457726886\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" Oct 30 18:24:50.486287 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:50.483033 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfbae34d-b472-4dd7-b862-dbb457726886-metrics-certs\") pod \"maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45\" (UID: \"bfbae34d-b472-4dd7-b862-dbb457726886\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" Oct 30 18:24:50.501465 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:50.495833 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4jrn\" (UniqueName: \"kubernetes.io/projected/bfbae34d-b472-4dd7-b862-dbb457726886-kube-api-access-l4jrn\") pod \"maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45\" (UID: \"bfbae34d-b472-4dd7-b862-dbb457726886\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" Oct 30 18:24:50.964650 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:50.964614 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfbae34d-b472-4dd7-b862-dbb457726886-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45\" (UID: \"bfbae34d-b472-4dd7-b862-dbb457726886\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" Oct 30 18:24:50.967647 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:50.967569 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfbae34d-b472-4dd7-b862-dbb457726886-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45\" (UID: \"bfbae34d-b472-4dd7-b862-dbb457726886\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" Oct 30 18:24:51.176223 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:51.176188 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" Oct 30 18:24:51.696351 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:51.695282 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d42172-54484dd44c-nn9fs" podUID="eb23ee45-fc23-4de0-9e81-71e087c0c502" containerName="payload-pre-processing" containerID="cri-o://23f349c98469c940bfc8bf68c577fa543e7e246cd650a07d5a1f9eb0d48d6155" gracePeriod=25 Oct 30 18:24:52.008285 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:52.008256 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d42172-54484dd44c-nn9fs" Oct 30 18:24:52.096849 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:52.090965 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb23ee45-fc23-4de0-9e81-71e087c0c502-kube-api-access-5mwdv\" (UniqueName: \"kubernetes.io/projected/eb23ee45-fc23-4de0-9e81-71e087c0c502-kube-api-access-5mwdv\") pod \"eb23ee45-fc23-4de0-9e81-71e087c0c502\" (UID: \"eb23ee45-fc23-4de0-9e81-71e087c0c502\") " Oct 30 18:24:52.096849 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:52.091217 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb23ee45-fc23-4de0-9e81-71e087c0c502-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb23ee45-fc23-4de0-9e81-71e087c0c502-plugins-config-volume\") pod \"eb23ee45-fc23-4de0-9e81-71e087c0c502\" (UID: \"eb23ee45-fc23-4de0-9e81-71e087c0c502\") " Oct 30 18:24:52.096849 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:52.093777 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb23ee45-fc23-4de0-9e81-71e087c0c502-plugins-config-volume" pod "eb23ee45-fc23-4de0-9e81-71e087c0c502" (UID: "eb23ee45-fc23-4de0-9e81-71e087c0c502"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:52.121459 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:52.121360 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb23ee45-fc23-4de0-9e81-71e087c0c502-kube-api-access-5mwdv" pod "eb23ee45-fc23-4de0-9e81-71e087c0c502" (UID: "eb23ee45-fc23-4de0-9e81-71e087c0c502"). InnerVolumeSpecName "kube-api-access-5mwdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:52.194335 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:52.194281 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mwdv\" (UniqueName: \"kubernetes.io/projected/eb23ee45-fc23-4de0-9e81-71e087c0c502-kube-api-access-5mwdv\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:24:52.194604 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:52.194384 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb23ee45-fc23-4de0-9e81-71e087c0c502-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:24:52.238018 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:52.237984 4455 generic.go:356] "Generic (PLEG): container finished" podID="eb23ee45-fc23-4de0-9e81-71e087c0c502" containerID="23f349c98469c940bfc8bf68c577fa543e7e246cd650a07d5a1f9eb0d48d6155" exitCode=0 Oct 30 18:24:52.238197 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:52.238036 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d42172-54484dd44c-nn9fs" Oct 30 18:24:52.238197 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:52.238064 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d42172-54484dd44c-nn9fs" event={"ID":"eb23ee45-fc23-4de0-9e81-71e087c0c502","Type":"ContainerDied","Data":"23f349c98469c940bfc8bf68c577fa543e7e246cd650a07d5a1f9eb0d48d6155"} Oct 30 18:24:52.238197 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:52.238117 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d42172-54484dd44c-nn9fs" event={"ID":"eb23ee45-fc23-4de0-9e81-71e087c0c502","Type":"ContainerDied","Data":"28a93fac0edb6407416b017b5befbc481c74431dd4b1b0e486cb0cdcd8170871"} Oct 30 18:24:52.238197 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:52.238138 4455 scope.go:122] "RemoveContainer" containerID="23f349c98469c940bfc8bf68c577fa543e7e246cd650a07d5a1f9eb0d48d6155" Oct 30 18:24:52.249743 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:52.249720 4455 scope.go:122] "RemoveContainer" containerID="23f349c98469c940bfc8bf68c577fa543e7e246cd650a07d5a1f9eb0d48d6155" Oct 30 18:24:52.250020 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:24:52.250003 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23f349c98469c940bfc8bf68c577fa543e7e246cd650a07d5a1f9eb0d48d6155\": container with ID starting with 23f349c98469c940bfc8bf68c577fa543e7e246cd650a07d5a1f9eb0d48d6155 not found: ID does not exist" containerID="23f349c98469c940bfc8bf68c577fa543e7e246cd650a07d5a1f9eb0d48d6155" Oct 30 18:24:52.250071 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:52.250029 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23f349c98469c940bfc8bf68c577fa543e7e246cd650a07d5a1f9eb0d48d6155"} err="failed to get container status \"23f349c98469c940bfc8bf68c577fa543e7e246cd650a07d5a1f9eb0d48d6155\": rpc error: code = NotFound desc = could not find container \"23f349c98469c940bfc8bf68c577fa543e7e246cd650a07d5a1f9eb0d48d6155\": container with ID starting with 23f349c98469c940bfc8bf68c577fa543e7e246cd650a07d5a1f9eb0d48d6155 not found: ID does not exist" Oct 30 18:24:52.298714 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:52.298684 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d42172-54484dd44c-nn9fs"] Oct 30 18:24:52.305564 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:52.305530 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d42172-54484dd44c-nn9fs"] Oct 30 18:24:53.688352 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:53.665511 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb23ee45-fc23-4de0-9e81-71e087c0c502" path="/var/lib/kubelet/pods/eb23ee45-fc23-4de0-9e81-71e087c0c502/volumes" Oct 30 18:24:53.780796 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:24:53.780754 4455 log.go:32] "RunPodSandbox from runtime service failed" err=< Oct 30 18:24:53.780796 ip-10-0-130-84 kubenswrapper[4455]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45_odh-ai-gateway-infra_bfbae34d-b472-4dd7-b862-dbb457726886_0(2aa0e9f6c530f82fbf0f086aec1e0c1ac158b78cc6de0cdf145e1044d0541422): error adding pod odh-ai-gateway-infra_maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"2aa0e9f6c530f82fbf0f086aec1e0c1ac158b78cc6de0cdf145e1044d0541422" Netns:"/var/run/netns/4235175a-6557-48ae-88ee-de958af04404" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45;K8S_POD_INFRA_CONTAINER_ID=2aa0e9f6c530f82fbf0f086aec1e0c1ac158b78cc6de0cdf145e1044d0541422;K8S_POD_UID=bfbae34d-b472-4dd7-b862-dbb457726886" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45/bfbae34d-b472-4dd7-b862-dbb457726886]: error waiting for pod: Get "https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45?timeout=1m0s": context deadline exceeded Oct 30 18:24:53.780796 ip-10-0-130-84 kubenswrapper[4455]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 18:24:53.780796 ip-10-0-130-84 kubenswrapper[4455]: > Oct 30 18:24:53.781012 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:24:53.780813 4455 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Oct 30 18:24:53.781012 ip-10-0-130-84 kubenswrapper[4455]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45_odh-ai-gateway-infra_bfbae34d-b472-4dd7-b862-dbb457726886_0(2aa0e9f6c530f82fbf0f086aec1e0c1ac158b78cc6de0cdf145e1044d0541422): error adding pod odh-ai-gateway-infra_maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"2aa0e9f6c530f82fbf0f086aec1e0c1ac158b78cc6de0cdf145e1044d0541422" Netns:"/var/run/netns/4235175a-6557-48ae-88ee-de958af04404" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45;K8S_POD_INFRA_CONTAINER_ID=2aa0e9f6c530f82fbf0f086aec1e0c1ac158b78cc6de0cdf145e1044d0541422;K8S_POD_UID=bfbae34d-b472-4dd7-b862-dbb457726886" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45/bfbae34d-b472-4dd7-b862-dbb457726886]: error waiting for pod: Get "https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45?timeout=1m0s": context deadline exceeded Oct 30 18:24:53.781012 ip-10-0-130-84 kubenswrapper[4455]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 18:24:53.781012 ip-10-0-130-84 kubenswrapper[4455]: > pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" Oct 30 18:24:53.781012 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:24:53.780831 4455 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=< Oct 30 18:24:53.781012 ip-10-0-130-84 kubenswrapper[4455]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45_odh-ai-gateway-infra_bfbae34d-b472-4dd7-b862-dbb457726886_0(2aa0e9f6c530f82fbf0f086aec1e0c1ac158b78cc6de0cdf145e1044d0541422): error adding pod odh-ai-gateway-infra_maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"2aa0e9f6c530f82fbf0f086aec1e0c1ac158b78cc6de0cdf145e1044d0541422" Netns:"/var/run/netns/4235175a-6557-48ae-88ee-de958af04404" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45;K8S_POD_INFRA_CONTAINER_ID=2aa0e9f6c530f82fbf0f086aec1e0c1ac158b78cc6de0cdf145e1044d0541422;K8S_POD_UID=bfbae34d-b472-4dd7-b862-dbb457726886" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45/bfbae34d-b472-4dd7-b862-dbb457726886]: error waiting for pod: Get "https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45?timeout=1m0s": context deadline exceeded Oct 30 18:24:53.781012 ip-10-0-130-84 kubenswrapper[4455]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 18:24:53.781012 ip-10-0-130-84 kubenswrapper[4455]: > pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" Oct 30 18:24:53.781012 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:24:53.780904 4455 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45_odh-ai-gateway-infra(bfbae34d-b472-4dd7-b862-dbb457726886)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45_odh-ai-gateway-infra(bfbae34d-b472-4dd7-b862-dbb457726886)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45_odh-ai-gateway-infra_bfbae34d-b472-4dd7-b862-dbb457726886_0(2aa0e9f6c530f82fbf0f086aec1e0c1ac158b78cc6de0cdf145e1044d0541422): error adding pod odh-ai-gateway-infra_maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"2aa0e9f6c530f82fbf0f086aec1e0c1ac158b78cc6de0cdf145e1044d0541422\\\" Netns:\\\"/var/run/netns/4235175a-6557-48ae-88ee-de958af04404\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45;K8S_POD_INFRA_CONTAINER_ID=2aa0e9f6c530f82fbf0f086aec1e0c1ac158b78cc6de0cdf145e1044d0541422;K8S_POD_UID=bfbae34d-b472-4dd7-b862-dbb457726886\\\" Path:\\\"\\\" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45/bfbae34d-b472-4dd7-b862-dbb457726886]: error waiting for pod: Get \\\"https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" podUID="bfbae34d-b472-4dd7-b862-dbb457726886" Oct 30 18:24:54.254697 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:54.254659 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" Oct 30 18:24:54.255063 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:54.255046 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" Oct 30 18:24:56.861029 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:24:56.860996 4455 log.go:32] "RunPodSandbox from runtime service failed" err=< Oct 30 18:24:56.861029 ip-10-0-130-84 kubenswrapper[4455]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45_odh-ai-gateway-infra_bfbae34d-b472-4dd7-b862-dbb457726886_0(76cb872e5ef0e9a55d05808a9656d79a6f2151724088190e3ccbe2be01f4a81c): error adding pod odh-ai-gateway-infra_maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"76cb872e5ef0e9a55d05808a9656d79a6f2151724088190e3ccbe2be01f4a81c" Netns:"/var/run/netns/a90ff3d1-87ad-429f-b120-07e425fe16a5" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45;K8S_POD_INFRA_CONTAINER_ID=76cb872e5ef0e9a55d05808a9656d79a6f2151724088190e3ccbe2be01f4a81c;K8S_POD_UID=bfbae34d-b472-4dd7-b862-dbb457726886" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45/bfbae34d-b472-4dd7-b862-dbb457726886]: error waiting for pod: Get "https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45?timeout=1m0s": context deadline exceeded Oct 30 18:24:56.861029 ip-10-0-130-84 kubenswrapper[4455]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 18:24:56.861029 ip-10-0-130-84 kubenswrapper[4455]: > Oct 30 18:24:56.861469 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:24:56.861049 4455 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Oct 30 18:24:56.861469 ip-10-0-130-84 kubenswrapper[4455]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45_odh-ai-gateway-infra_bfbae34d-b472-4dd7-b862-dbb457726886_0(76cb872e5ef0e9a55d05808a9656d79a6f2151724088190e3ccbe2be01f4a81c): error adding pod odh-ai-gateway-infra_maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"76cb872e5ef0e9a55d05808a9656d79a6f2151724088190e3ccbe2be01f4a81c" Netns:"/var/run/netns/a90ff3d1-87ad-429f-b120-07e425fe16a5" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45;K8S_POD_INFRA_CONTAINER_ID=76cb872e5ef0e9a55d05808a9656d79a6f2151724088190e3ccbe2be01f4a81c;K8S_POD_UID=bfbae34d-b472-4dd7-b862-dbb457726886" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45/bfbae34d-b472-4dd7-b862-dbb457726886]: error waiting for pod: Get "https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45?timeout=1m0s": context deadline exceeded Oct 30 18:24:56.861469 ip-10-0-130-84 kubenswrapper[4455]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 18:24:56.861469 ip-10-0-130-84 kubenswrapper[4455]: > pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" Oct 30 18:24:56.861469 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:24:56.861066 4455 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=< Oct 30 18:24:56.861469 ip-10-0-130-84 kubenswrapper[4455]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45_odh-ai-gateway-infra_bfbae34d-b472-4dd7-b862-dbb457726886_0(76cb872e5ef0e9a55d05808a9656d79a6f2151724088190e3ccbe2be01f4a81c): error adding pod odh-ai-gateway-infra_maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"76cb872e5ef0e9a55d05808a9656d79a6f2151724088190e3ccbe2be01f4a81c" Netns:"/var/run/netns/a90ff3d1-87ad-429f-b120-07e425fe16a5" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45;K8S_POD_INFRA_CONTAINER_ID=76cb872e5ef0e9a55d05808a9656d79a6f2151724088190e3ccbe2be01f4a81c;K8S_POD_UID=bfbae34d-b472-4dd7-b862-dbb457726886" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45/bfbae34d-b472-4dd7-b862-dbb457726886]: error waiting for pod: Get "https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45?timeout=1m0s": context deadline exceeded Oct 30 18:24:56.861469 ip-10-0-130-84 kubenswrapper[4455]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 18:24:56.861469 ip-10-0-130-84 kubenswrapper[4455]: > pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" Oct 30 18:24:56.861469 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:24:56.861151 4455 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45_odh-ai-gateway-infra(bfbae34d-b472-4dd7-b862-dbb457726886)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45_odh-ai-gateway-infra(bfbae34d-b472-4dd7-b862-dbb457726886)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45_odh-ai-gateway-infra_bfbae34d-b472-4dd7-b862-dbb457726886_0(76cb872e5ef0e9a55d05808a9656d79a6f2151724088190e3ccbe2be01f4a81c): error adding pod odh-ai-gateway-infra_maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"76cb872e5ef0e9a55d05808a9656d79a6f2151724088190e3ccbe2be01f4a81c\\\" Netns:\\\"/var/run/netns/a90ff3d1-87ad-429f-b120-07e425fe16a5\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45;K8S_POD_INFRA_CONTAINER_ID=76cb872e5ef0e9a55d05808a9656d79a6f2151724088190e3ccbe2be01f4a81c;K8S_POD_UID=bfbae34d-b472-4dd7-b862-dbb457726886\\\" Path:\\\"\\\" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45/bfbae34d-b472-4dd7-b862-dbb457726886]: error waiting for pod: Get \\\"https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" podUID="bfbae34d-b472-4dd7-b862-dbb457726886" Oct 30 18:24:57.259995 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:57.259955 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch"] Oct 30 18:24:57.260287 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:57.260251 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch" podUID="00eedd3e-a830-4069-a3fd-4c43c149b943" containerName="maas-api" containerID="cri-o://9e6597864e5b0562f710b2455bf03a58cca8760c32508aee8482c4b8fc46192f" gracePeriod=30 Oct 30 18:24:57.754807 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:57.754770 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch" Oct 30 18:24:57.883347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:57.879421 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00eedd3e-a830-4069-a3fd-4c43c149b943-metrics-certs\" (UniqueName: \"kubernetes.io/secret/00eedd3e-a830-4069-a3fd-4c43c149b943-metrics-certs\") pod \"00eedd3e-a830-4069-a3fd-4c43c149b943\" (UID: \"00eedd3e-a830-4069-a3fd-4c43c149b943\") " Oct 30 18:24:57.883347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:57.879579 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00eedd3e-a830-4069-a3fd-4c43c149b943-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00eedd3e-a830-4069-a3fd-4c43c149b943-maas-api-tls\") pod \"00eedd3e-a830-4069-a3fd-4c43c149b943\" (UID: \"00eedd3e-a830-4069-a3fd-4c43c149b943\") " Oct 30 18:24:57.883347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:57.879628 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00eedd3e-a830-4069-a3fd-4c43c149b943-kube-api-access-cfn87\" (UniqueName: \"kubernetes.io/projected/00eedd3e-a830-4069-a3fd-4c43c149b943-kube-api-access-cfn87\") pod \"00eedd3e-a830-4069-a3fd-4c43c149b943\" (UID: \"00eedd3e-a830-4069-a3fd-4c43c149b943\") " Oct 30 18:24:57.883347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:57.879716 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00eedd3e-a830-4069-a3fd-4c43c149b943-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00eedd3e-a830-4069-a3fd-4c43c149b943-openshift-service-ca\") pod \"00eedd3e-a830-4069-a3fd-4c43c149b943\" (UID: \"00eedd3e-a830-4069-a3fd-4c43c149b943\") " Oct 30 18:24:57.883347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:57.881508 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00eedd3e-a830-4069-a3fd-4c43c149b943-openshift-service-ca" pod "00eedd3e-a830-4069-a3fd-4c43c149b943" (UID: "00eedd3e-a830-4069-a3fd-4c43c149b943"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:57.885284 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:57.885251 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00eedd3e-a830-4069-a3fd-4c43c149b943-maas-api-tls" pod "00eedd3e-a830-4069-a3fd-4c43c149b943" (UID: "00eedd3e-a830-4069-a3fd-4c43c149b943"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:24:57.932942 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:57.930935 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00eedd3e-a830-4069-a3fd-4c43c149b943-metrics-certs" pod "00eedd3e-a830-4069-a3fd-4c43c149b943" (UID: "00eedd3e-a830-4069-a3fd-4c43c149b943"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:24:57.932942 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:57.931215 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00eedd3e-a830-4069-a3fd-4c43c149b943-kube-api-access-cfn87" pod "00eedd3e-a830-4069-a3fd-4c43c149b943" (UID: "00eedd3e-a830-4069-a3fd-4c43c149b943"). InnerVolumeSpecName "kube-api-access-cfn87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:57.987358 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:57.987013 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00eedd3e-a830-4069-a3fd-4c43c149b943-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:24:57.987358 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:57.987038 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00eedd3e-a830-4069-a3fd-4c43c149b943-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:24:57.987358 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:57.987048 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfn87\" (UniqueName: \"kubernetes.io/projected/00eedd3e-a830-4069-a3fd-4c43c149b943-kube-api-access-cfn87\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:24:57.987358 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:57.987057 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00eedd3e-a830-4069-a3fd-4c43c149b943-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:24:58.319085 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:58.319045 4455 generic.go:356] "Generic (PLEG): container finished" podID="00eedd3e-a830-4069-a3fd-4c43c149b943" containerID="9e6597864e5b0562f710b2455bf03a58cca8760c32508aee8482c4b8fc46192f" exitCode=0 Oct 30 18:24:58.320184 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:58.319410 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch" event={"ID":"00eedd3e-a830-4069-a3fd-4c43c149b943","Type":"ContainerDied","Data":"9e6597864e5b0562f710b2455bf03a58cca8760c32508aee8482c4b8fc46192f"} Oct 30 18:24:58.320184 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:58.319477 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch" event={"ID":"00eedd3e-a830-4069-a3fd-4c43c149b943","Type":"ContainerDied","Data":"526474a3d9cf60ec820bdea1f6765cb4f342650cc157e8acc06dc66e9aa68ebc"} Oct 30 18:24:58.320184 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:58.319510 4455 scope.go:122] "RemoveContainer" containerID="9e6597864e5b0562f710b2455bf03a58cca8760c32508aee8482c4b8fc46192f" Oct 30 18:24:58.320184 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:58.319645 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch" Oct 30 18:24:58.345309 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:58.345265 4455 scope.go:122] "RemoveContainer" containerID="59babe6be3c2d3a6c62bc74301f084b559b1746d5a0bffd99388015e47a3bb3b" Oct 30 18:24:58.358630 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:58.358605 4455 scope.go:122] "RemoveContainer" containerID="9e6597864e5b0562f710b2455bf03a58cca8760c32508aee8482c4b8fc46192f" Oct 30 18:24:58.358962 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:24:58.358937 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e6597864e5b0562f710b2455bf03a58cca8760c32508aee8482c4b8fc46192f\": container with ID starting with 9e6597864e5b0562f710b2455bf03a58cca8760c32508aee8482c4b8fc46192f not found: ID does not exist" containerID="9e6597864e5b0562f710b2455bf03a58cca8760c32508aee8482c4b8fc46192f" Oct 30 18:24:58.359008 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:58.358973 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e6597864e5b0562f710b2455bf03a58cca8760c32508aee8482c4b8fc46192f"} err="failed to get container status \"9e6597864e5b0562f710b2455bf03a58cca8760c32508aee8482c4b8fc46192f\": rpc error: code = NotFound desc = could not find container \"9e6597864e5b0562f710b2455bf03a58cca8760c32508aee8482c4b8fc46192f\": container with ID starting with 9e6597864e5b0562f710b2455bf03a58cca8760c32508aee8482c4b8fc46192f not found: ID does not exist" Oct 30 18:24:58.359008 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:58.358992 4455 scope.go:122] "RemoveContainer" containerID="59babe6be3c2d3a6c62bc74301f084b559b1746d5a0bffd99388015e47a3bb3b" Oct 30 18:24:58.359337 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:24:58.359301 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59babe6be3c2d3a6c62bc74301f084b559b1746d5a0bffd99388015e47a3bb3b\": container with ID starting with 59babe6be3c2d3a6c62bc74301f084b559b1746d5a0bffd99388015e47a3bb3b not found: ID does not exist" containerID="59babe6be3c2d3a6c62bc74301f084b559b1746d5a0bffd99388015e47a3bb3b" Oct 30 18:24:58.359416 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:58.359347 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59babe6be3c2d3a6c62bc74301f084b559b1746d5a0bffd99388015e47a3bb3b"} err="failed to get container status \"59babe6be3c2d3a6c62bc74301f084b559b1746d5a0bffd99388015e47a3bb3b\": rpc error: code = NotFound desc = could not find container \"59babe6be3c2d3a6c62bc74301f084b559b1746d5a0bffd99388015e47a3bb3b\": container with ID starting with 59babe6be3c2d3a6c62bc74301f084b559b1746d5a0bffd99388015e47a3bb3b not found: ID does not exist" Oct 30 18:24:58.393993 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:58.393955 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch"] Oct 30 18:24:58.409191 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:58.408732 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d42172-66d45cbf5-mzqch"] Oct 30 18:24:58.731080 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:58.731045 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9"] Oct 30 18:24:58.731856 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:58.731841 4455 cpu_manager.go:414] "containerMap: removing container" podUID="00eedd3e-a830-4069-a3fd-4c43c149b943" containerName="maas-api" Oct 30 18:24:58.731907 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:58.731858 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00eedd3e-a830-4069-a3fd-4c43c149b943" containerName="maas-api" Oct 30 18:24:58.731907 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:58.731870 4455 cpu_manager.go:414] "containerMap: removing container" podUID="eb23ee45-fc23-4de0-9e81-71e087c0c502" containerName="payload-pre-processing" Oct 30 18:24:58.731907 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:58.731875 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb23ee45-fc23-4de0-9e81-71e087c0c502" containerName="payload-pre-processing" Oct 30 18:24:58.731907 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:58.731889 4455 cpu_manager.go:414] "containerMap: removing container" podUID="00eedd3e-a830-4069-a3fd-4c43c149b943" containerName="maas-api" Oct 30 18:24:58.731907 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:58.731893 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00eedd3e-a830-4069-a3fd-4c43c149b943" containerName="maas-api" Oct 30 18:24:58.732077 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:58.732067 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="00eedd3e-a830-4069-a3fd-4c43c149b943" containerName="maas-api" Oct 30 18:24:58.732114 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:58.732078 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb23ee45-fc23-4de0-9e81-71e087c0c502" containerName="payload-pre-processing" Oct 30 18:24:58.732114 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:58.732086 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="00eedd3e-a830-4069-a3fd-4c43c149b943" containerName="maas-api" Oct 30 18:24:58.738501 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:58.738472 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9" Oct 30 18:24:58.755963 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:58.755921 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-42fad7\"" Oct 30 18:24:58.756176 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:58.756006 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-42fad7-dockercfg-m65rm\"" Oct 30 18:24:58.759492 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:58.759463 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9"] Oct 30 18:24:58.861550 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:58.860215 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qb4k\" (UniqueName: \"kubernetes.io/projected/46eee9bc-87c0-4421-92e7-547a494dbf3c-kube-api-access-4qb4k\") pod \"payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9\" (UID: \"46eee9bc-87c0-4421-92e7-547a494dbf3c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9" Oct 30 18:24:58.861550 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:58.860413 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46eee9bc-87c0-4421-92e7-547a494dbf3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9\" (UID: \"46eee9bc-87c0-4421-92e7-547a494dbf3c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9" Oct 30 18:24:58.865423 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:58.865386 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl"] Oct 30 18:24:59.026681 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:59.025427 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46eee9bc-87c0-4421-92e7-547a494dbf3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9\" (UID: \"46eee9bc-87c0-4421-92e7-547a494dbf3c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9" Oct 30 18:24:59.026681 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:59.025764 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qb4k\" (UniqueName: \"kubernetes.io/projected/46eee9bc-87c0-4421-92e7-547a494dbf3c-kube-api-access-4qb4k\") pod \"payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9\" (UID: \"46eee9bc-87c0-4421-92e7-547a494dbf3c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9" Oct 30 18:24:59.037175 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:59.036034 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46eee9bc-87c0-4421-92e7-547a494dbf3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9\" (UID: \"46eee9bc-87c0-4421-92e7-547a494dbf3c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9" Oct 30 18:24:59.047348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:59.046618 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl"] Oct 30 18:24:59.047348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:59.046736 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl" Oct 30 18:24:59.097366 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:59.097310 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qb4k\" (UniqueName: \"kubernetes.io/projected/46eee9bc-87c0-4421-92e7-547a494dbf3c-kube-api-access-4qb4k\") pod \"payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9\" (UID: \"46eee9bc-87c0-4421-92e7-547a494dbf3c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9" Oct 30 18:24:59.129792 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:59.129472 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a166721-a635-4df9-adfe-7887eb5e7b39-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl\" (UID: \"6a166721-a635-4df9-adfe-7887eb5e7b39\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl" Oct 30 18:24:59.129972 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:59.129792 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spq86\" (UniqueName: \"kubernetes.io/projected/6a166721-a635-4df9-adfe-7887eb5e7b39-kube-api-access-spq86\") pod \"payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl\" (UID: \"6a166721-a635-4df9-adfe-7887eb5e7b39\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl" Oct 30 18:24:59.231251 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:59.231214 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a166721-a635-4df9-adfe-7887eb5e7b39-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl\" (UID: \"6a166721-a635-4df9-adfe-7887eb5e7b39\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl" Oct 30 18:24:59.231470 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:59.231381 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spq86\" (UniqueName: \"kubernetes.io/projected/6a166721-a635-4df9-adfe-7887eb5e7b39-kube-api-access-spq86\") pod \"payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl\" (UID: \"6a166721-a635-4df9-adfe-7887eb5e7b39\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl" Oct 30 18:24:59.232169 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:59.232149 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a166721-a635-4df9-adfe-7887eb5e7b39-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl\" (UID: \"6a166721-a635-4df9-adfe-7887eb5e7b39\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl" Oct 30 18:24:59.256977 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:59.256945 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spq86\" (UniqueName: \"kubernetes.io/projected/6a166721-a635-4df9-adfe-7887eb5e7b39-kube-api-access-spq86\") pod \"payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl\" (UID: \"6a166721-a635-4df9-adfe-7887eb5e7b39\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl" Oct 30 18:24:59.354760 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:59.354662 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9" Oct 30 18:24:59.445110 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:59.445066 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl" Oct 30 18:24:59.652164 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:24:59.652074 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00eedd3e-a830-4069-a3fd-4c43c149b943" path="/var/lib/kubelet/pods/00eedd3e-a830-4069-a3fd-4c43c149b943/volumes" Oct 30 18:25:00.392995 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:25:00.392926 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:25:01.956590 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:25:01.956544 4455 log.go:32] "RunPodSandbox from runtime service failed" err=< Oct 30 18:25:01.956590 ip-10-0-130-84 kubenswrapper[4455]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9_openshift-ingress_46eee9bc-87c0-4421-92e7-547a494dbf3c_0(d75d960a160af4bea03dcedd145f128706b40b551b94c8fe91931bd6bd7f6e2c): error adding pod openshift-ingress_payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"d75d960a160af4bea03dcedd145f128706b40b551b94c8fe91931bd6bd7f6e2c" Netns:"/var/run/netns/e7ebec11-2073-424b-9fd8-33762f48750e" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9;K8S_POD_INFRA_CONTAINER_ID=d75d960a160af4bea03dcedd145f128706b40b551b94c8fe91931bd6bd7f6e2c;K8S_POD_UID=46eee9bc-87c0-4421-92e7-547a494dbf3c" Path:"" ERRORED: error configuring pod [openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9] networking: Multus: [openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9/46eee9bc-87c0-4421-92e7-547a494dbf3c]: error waiting for pod: Get "https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9?timeout=1m0s": context deadline exceeded Oct 30 18:25:01.956590 ip-10-0-130-84 kubenswrapper[4455]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 18:25:01.956590 ip-10-0-130-84 kubenswrapper[4455]: > Oct 30 18:25:01.957190 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:25:01.956617 4455 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Oct 30 18:25:01.957190 ip-10-0-130-84 kubenswrapper[4455]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9_openshift-ingress_46eee9bc-87c0-4421-92e7-547a494dbf3c_0(d75d960a160af4bea03dcedd145f128706b40b551b94c8fe91931bd6bd7f6e2c): error adding pod openshift-ingress_payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"d75d960a160af4bea03dcedd145f128706b40b551b94c8fe91931bd6bd7f6e2c" Netns:"/var/run/netns/e7ebec11-2073-424b-9fd8-33762f48750e" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9;K8S_POD_INFRA_CONTAINER_ID=d75d960a160af4bea03dcedd145f128706b40b551b94c8fe91931bd6bd7f6e2c;K8S_POD_UID=46eee9bc-87c0-4421-92e7-547a494dbf3c" Path:"" ERRORED: error configuring pod [openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9] networking: Multus: [openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9/46eee9bc-87c0-4421-92e7-547a494dbf3c]: error waiting for pod: Get "https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9?timeout=1m0s": context deadline exceeded Oct 30 18:25:01.957190 ip-10-0-130-84 kubenswrapper[4455]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 18:25:01.957190 ip-10-0-130-84 kubenswrapper[4455]: > pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9" Oct 30 18:25:01.957190 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:25:01.956641 4455 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=< Oct 30 18:25:01.957190 ip-10-0-130-84 kubenswrapper[4455]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9_openshift-ingress_46eee9bc-87c0-4421-92e7-547a494dbf3c_0(d75d960a160af4bea03dcedd145f128706b40b551b94c8fe91931bd6bd7f6e2c): error adding pod openshift-ingress_payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"d75d960a160af4bea03dcedd145f128706b40b551b94c8fe91931bd6bd7f6e2c" Netns:"/var/run/netns/e7ebec11-2073-424b-9fd8-33762f48750e" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9;K8S_POD_INFRA_CONTAINER_ID=d75d960a160af4bea03dcedd145f128706b40b551b94c8fe91931bd6bd7f6e2c;K8S_POD_UID=46eee9bc-87c0-4421-92e7-547a494dbf3c" Path:"" ERRORED: error configuring pod [openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9] networking: Multus: [openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9/46eee9bc-87c0-4421-92e7-547a494dbf3c]: error waiting for pod: Get "https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9?timeout=1m0s": context deadline exceeded Oct 30 18:25:01.957190 ip-10-0-130-84 kubenswrapper[4455]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 18:25:01.957190 ip-10-0-130-84 kubenswrapper[4455]: > pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9" Oct 30 18:25:01.957190 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:25:01.956776 4455 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9_openshift-ingress(46eee9bc-87c0-4421-92e7-547a494dbf3c)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9_openshift-ingress(46eee9bc-87c0-4421-92e7-547a494dbf3c)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9_openshift-ingress_46eee9bc-87c0-4421-92e7-547a494dbf3c_0(d75d960a160af4bea03dcedd145f128706b40b551b94c8fe91931bd6bd7f6e2c): error adding pod openshift-ingress_payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"d75d960a160af4bea03dcedd145f128706b40b551b94c8fe91931bd6bd7f6e2c\\\" Netns:\\\"/var/run/netns/e7ebec11-2073-424b-9fd8-33762f48750e\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9;K8S_POD_INFRA_CONTAINER_ID=d75d960a160af4bea03dcedd145f128706b40b551b94c8fe91931bd6bd7f6e2c;K8S_POD_UID=46eee9bc-87c0-4421-92e7-547a494dbf3c\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9] networking: Multus: [openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9/46eee9bc-87c0-4421-92e7-547a494dbf3c]: error waiting for pod: Get \\\"https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9" podUID="46eee9bc-87c0-4421-92e7-547a494dbf3c" Oct 30 18:25:02.052881 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:25:02.052808 4455 log.go:32] "RunPodSandbox from runtime service failed" err=< Oct 30 18:25:02.052881 ip-10-0-130-84 kubenswrapper[4455]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl_openshift-ingress_6a166721-a635-4df9-adfe-7887eb5e7b39_0(e456e9e15295ab889301780c08c609e811a99205ef2bfe0faba296e12e30cfa1): error adding pod openshift-ingress_payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"e456e9e15295ab889301780c08c609e811a99205ef2bfe0faba296e12e30cfa1" Netns:"/var/run/netns/fd269dd5-e2eb-4dc8-8e24-9efa79cb7d1d" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl;K8S_POD_INFRA_CONTAINER_ID=e456e9e15295ab889301780c08c609e811a99205ef2bfe0faba296e12e30cfa1;K8S_POD_UID=6a166721-a635-4df9-adfe-7887eb5e7b39" Path:"" ERRORED: error configuring pod [openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl] networking: Multus: [openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl/6a166721-a635-4df9-adfe-7887eb5e7b39]: error waiting for pod: Get "https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl?timeout=1m0s": context deadline exceeded Oct 30 18:25:02.052881 ip-10-0-130-84 kubenswrapper[4455]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 18:25:02.052881 ip-10-0-130-84 kubenswrapper[4455]: > Oct 30 18:25:02.053115 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:25:02.052876 4455 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Oct 30 18:25:02.053115 ip-10-0-130-84 kubenswrapper[4455]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl_openshift-ingress_6a166721-a635-4df9-adfe-7887eb5e7b39_0(e456e9e15295ab889301780c08c609e811a99205ef2bfe0faba296e12e30cfa1): error adding pod openshift-ingress_payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"e456e9e15295ab889301780c08c609e811a99205ef2bfe0faba296e12e30cfa1" Netns:"/var/run/netns/fd269dd5-e2eb-4dc8-8e24-9efa79cb7d1d" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl;K8S_POD_INFRA_CONTAINER_ID=e456e9e15295ab889301780c08c609e811a99205ef2bfe0faba296e12e30cfa1;K8S_POD_UID=6a166721-a635-4df9-adfe-7887eb5e7b39" Path:"" ERRORED: error configuring pod [openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl] networking: Multus: [openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl/6a166721-a635-4df9-adfe-7887eb5e7b39]: error waiting for pod: Get "https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl?timeout=1m0s": context deadline exceeded Oct 30 18:25:02.053115 ip-10-0-130-84 kubenswrapper[4455]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 18:25:02.053115 ip-10-0-130-84 kubenswrapper[4455]: > pod="openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl" Oct 30 18:25:02.053115 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:25:02.052901 4455 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=< Oct 30 18:25:02.053115 ip-10-0-130-84 kubenswrapper[4455]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl_openshift-ingress_6a166721-a635-4df9-adfe-7887eb5e7b39_0(e456e9e15295ab889301780c08c609e811a99205ef2bfe0faba296e12e30cfa1): error adding pod openshift-ingress_payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"e456e9e15295ab889301780c08c609e811a99205ef2bfe0faba296e12e30cfa1" Netns:"/var/run/netns/fd269dd5-e2eb-4dc8-8e24-9efa79cb7d1d" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl;K8S_POD_INFRA_CONTAINER_ID=e456e9e15295ab889301780c08c609e811a99205ef2bfe0faba296e12e30cfa1;K8S_POD_UID=6a166721-a635-4df9-adfe-7887eb5e7b39" Path:"" ERRORED: error configuring pod [openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl] networking: Multus: [openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl/6a166721-a635-4df9-adfe-7887eb5e7b39]: error waiting for pod: Get "https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl?timeout=1m0s": context deadline exceeded Oct 30 18:25:02.053115 ip-10-0-130-84 kubenswrapper[4455]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 18:25:02.053115 ip-10-0-130-84 kubenswrapper[4455]: > pod="openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl" Oct 30 18:25:02.053115 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:25:02.052988 4455 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl_openshift-ingress(6a166721-a635-4df9-adfe-7887eb5e7b39)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl_openshift-ingress(6a166721-a635-4df9-adfe-7887eb5e7b39)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl_openshift-ingress_6a166721-a635-4df9-adfe-7887eb5e7b39_0(e456e9e15295ab889301780c08c609e811a99205ef2bfe0faba296e12e30cfa1): error adding pod openshift-ingress_payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"e456e9e15295ab889301780c08c609e811a99205ef2bfe0faba296e12e30cfa1\\\" Netns:\\\"/var/run/netns/fd269dd5-e2eb-4dc8-8e24-9efa79cb7d1d\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl;K8S_POD_INFRA_CONTAINER_ID=e456e9e15295ab889301780c08c609e811a99205ef2bfe0faba296e12e30cfa1;K8S_POD_UID=6a166721-a635-4df9-adfe-7887eb5e7b39\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl] networking: Multus: [openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl/6a166721-a635-4df9-adfe-7887eb5e7b39]: error waiting for pod: Get \\\"https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl" podUID="6a166721-a635-4df9-adfe-7887eb5e7b39" Oct 30 18:25:02.358114 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:02.358082 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9" Oct 30 18:25:02.358451 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:02.358437 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9" Oct 30 18:25:04.584808 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:04.584776 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b8b654-6c7556f4cb-8prx6"] Oct 30 18:25:04.624336 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:04.624298 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b8b654-78678d94b7-nm799"] Oct 30 18:25:04.626156 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:04.626137 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" Oct 30 18:25:04.626502 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:04.626490 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" Oct 30 18:25:04.985182 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:25:04.985144 4455 log.go:32] "RunPodSandbox from runtime service failed" err=< Oct 30 18:25:04.985182 ip-10-0-130-84 kubenswrapper[4455]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9_openshift-ingress_46eee9bc-87c0-4421-92e7-547a494dbf3c_0(f584d7d5632cae08389973a4dc0ca6e9595e42f08f317ca8747581adbd235f55): error adding pod openshift-ingress_payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"f584d7d5632cae08389973a4dc0ca6e9595e42f08f317ca8747581adbd235f55" Netns:"/var/run/netns/db94b451-ae31-4917-9102-e84913bcc7c8" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9;K8S_POD_INFRA_CONTAINER_ID=f584d7d5632cae08389973a4dc0ca6e9595e42f08f317ca8747581adbd235f55;K8S_POD_UID=46eee9bc-87c0-4421-92e7-547a494dbf3c" Path:"" ERRORED: error configuring pod [openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9] networking: Multus: [openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9/46eee9bc-87c0-4421-92e7-547a494dbf3c]: error waiting for pod: Get "https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9?timeout=1m0s": context deadline exceeded Oct 30 18:25:04.985182 ip-10-0-130-84 kubenswrapper[4455]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 18:25:04.985182 ip-10-0-130-84 kubenswrapper[4455]: > Oct 30 18:25:04.985532 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:25:04.985510 4455 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Oct 30 18:25:04.985532 ip-10-0-130-84 kubenswrapper[4455]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9_openshift-ingress_46eee9bc-87c0-4421-92e7-547a494dbf3c_0(f584d7d5632cae08389973a4dc0ca6e9595e42f08f317ca8747581adbd235f55): error adding pod openshift-ingress_payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"f584d7d5632cae08389973a4dc0ca6e9595e42f08f317ca8747581adbd235f55" Netns:"/var/run/netns/db94b451-ae31-4917-9102-e84913bcc7c8" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9;K8S_POD_INFRA_CONTAINER_ID=f584d7d5632cae08389973a4dc0ca6e9595e42f08f317ca8747581adbd235f55;K8S_POD_UID=46eee9bc-87c0-4421-92e7-547a494dbf3c" Path:"" ERRORED: error configuring pod [openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9] networking: Multus: [openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9/46eee9bc-87c0-4421-92e7-547a494dbf3c]: error waiting for pod: Get "https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9?timeout=1m0s": context deadline exceeded Oct 30 18:25:04.985532 ip-10-0-130-84 kubenswrapper[4455]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 18:25:04.985532 ip-10-0-130-84 kubenswrapper[4455]: > pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9" Oct 30 18:25:04.985781 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:25:04.985765 4455 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=< Oct 30 18:25:04.985781 ip-10-0-130-84 kubenswrapper[4455]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9_openshift-ingress_46eee9bc-87c0-4421-92e7-547a494dbf3c_0(f584d7d5632cae08389973a4dc0ca6e9595e42f08f317ca8747581adbd235f55): error adding pod openshift-ingress_payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"f584d7d5632cae08389973a4dc0ca6e9595e42f08f317ca8747581adbd235f55" Netns:"/var/run/netns/db94b451-ae31-4917-9102-e84913bcc7c8" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9;K8S_POD_INFRA_CONTAINER_ID=f584d7d5632cae08389973a4dc0ca6e9595e42f08f317ca8747581adbd235f55;K8S_POD_UID=46eee9bc-87c0-4421-92e7-547a494dbf3c" Path:"" ERRORED: error configuring pod [openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9] networking: Multus: [openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9/46eee9bc-87c0-4421-92e7-547a494dbf3c]: error waiting for pod: Get "https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9?timeout=1m0s": context deadline exceeded Oct 30 18:25:04.985781 ip-10-0-130-84 kubenswrapper[4455]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 18:25:04.985781 ip-10-0-130-84 kubenswrapper[4455]: > pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9" Oct 30 18:25:04.986133 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:25:04.986082 4455 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9_openshift-ingress(46eee9bc-87c0-4421-92e7-547a494dbf3c)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9_openshift-ingress(46eee9bc-87c0-4421-92e7-547a494dbf3c)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9_openshift-ingress_46eee9bc-87c0-4421-92e7-547a494dbf3c_0(f584d7d5632cae08389973a4dc0ca6e9595e42f08f317ca8747581adbd235f55): error adding pod openshift-ingress_payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"f584d7d5632cae08389973a4dc0ca6e9595e42f08f317ca8747581adbd235f55\\\" Netns:\\\"/var/run/netns/db94b451-ae31-4917-9102-e84913bcc7c8\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9;K8S_POD_INFRA_CONTAINER_ID=f584d7d5632cae08389973a4dc0ca6e9595e42f08f317ca8747581adbd235f55;K8S_POD_UID=46eee9bc-87c0-4421-92e7-547a494dbf3c\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9] networking: Multus: [openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9/46eee9bc-87c0-4421-92e7-547a494dbf3c]: error waiting for pod: Get \\\"https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9" podUID="46eee9bc-87c0-4421-92e7-547a494dbf3c" Oct 30 18:25:07.258067 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:25:07.258016 4455 log.go:32] "RunPodSandbox from runtime service failed" err=< Oct 30 18:25:07.258067 ip-10-0-130-84 kubenswrapper[4455]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7_odh-ai-gateway-infra_5e30a2f8-aa4d-4dea-8468-aa09e6100c58_0(ed009013f5e0cdb4cecdff219276392e7c109f83af64a2a4dff02496cef2d18c): error adding pod odh-ai-gateway-infra_maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"ed009013f5e0cdb4cecdff219276392e7c109f83af64a2a4dff02496cef2d18c" Netns:"/var/run/netns/83123419-6b94-4cac-b9c9-8d9c7838640c" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7;K8S_POD_INFRA_CONTAINER_ID=ed009013f5e0cdb4cecdff219276392e7c109f83af64a2a4dff02496cef2d18c;K8S_POD_UID=5e30a2f8-aa4d-4dea-8468-aa09e6100c58" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7/5e30a2f8-aa4d-4dea-8468-aa09e6100c58]: error waiting for pod: Get "https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7?timeout=1m0s": context deadline exceeded Oct 30 18:25:07.258067 ip-10-0-130-84 kubenswrapper[4455]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 18:25:07.258067 ip-10-0-130-84 kubenswrapper[4455]: > Oct 30 18:25:07.258812 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:25:07.258094 4455 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Oct 30 18:25:07.258812 ip-10-0-130-84 kubenswrapper[4455]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7_odh-ai-gateway-infra_5e30a2f8-aa4d-4dea-8468-aa09e6100c58_0(ed009013f5e0cdb4cecdff219276392e7c109f83af64a2a4dff02496cef2d18c): error adding pod odh-ai-gateway-infra_maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"ed009013f5e0cdb4cecdff219276392e7c109f83af64a2a4dff02496cef2d18c" Netns:"/var/run/netns/83123419-6b94-4cac-b9c9-8d9c7838640c" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7;K8S_POD_INFRA_CONTAINER_ID=ed009013f5e0cdb4cecdff219276392e7c109f83af64a2a4dff02496cef2d18c;K8S_POD_UID=5e30a2f8-aa4d-4dea-8468-aa09e6100c58" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7/5e30a2f8-aa4d-4dea-8468-aa09e6100c58]: error waiting for pod: Get "https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7?timeout=1m0s": context deadline exceeded Oct 30 18:25:07.258812 ip-10-0-130-84 kubenswrapper[4455]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 18:25:07.258812 ip-10-0-130-84 kubenswrapper[4455]: > pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" Oct 30 18:25:07.258812 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:25:07.258119 4455 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=< Oct 30 18:25:07.258812 ip-10-0-130-84 kubenswrapper[4455]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7_odh-ai-gateway-infra_5e30a2f8-aa4d-4dea-8468-aa09e6100c58_0(ed009013f5e0cdb4cecdff219276392e7c109f83af64a2a4dff02496cef2d18c): error adding pod odh-ai-gateway-infra_maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"ed009013f5e0cdb4cecdff219276392e7c109f83af64a2a4dff02496cef2d18c" Netns:"/var/run/netns/83123419-6b94-4cac-b9c9-8d9c7838640c" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7;K8S_POD_INFRA_CONTAINER_ID=ed009013f5e0cdb4cecdff219276392e7c109f83af64a2a4dff02496cef2d18c;K8S_POD_UID=5e30a2f8-aa4d-4dea-8468-aa09e6100c58" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7/5e30a2f8-aa4d-4dea-8468-aa09e6100c58]: error waiting for pod: Get "https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7?timeout=1m0s": context deadline exceeded Oct 30 18:25:07.258812 ip-10-0-130-84 kubenswrapper[4455]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Oct 30 18:25:07.258812 ip-10-0-130-84 kubenswrapper[4455]: > pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" Oct 30 18:25:07.258812 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:25:07.258226 4455 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7_odh-ai-gateway-infra(5e30a2f8-aa4d-4dea-8468-aa09e6100c58)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7_odh-ai-gateway-infra(5e30a2f8-aa4d-4dea-8468-aa09e6100c58)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7_odh-ai-gateway-infra_5e30a2f8-aa4d-4dea-8468-aa09e6100c58_0(ed009013f5e0cdb4cecdff219276392e7c109f83af64a2a4dff02496cef2d18c): error adding pod odh-ai-gateway-infra_maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"ed009013f5e0cdb4cecdff219276392e7c109f83af64a2a4dff02496cef2d18c\\\" Netns:\\\"/var/run/netns/83123419-6b94-4cac-b9c9-8d9c7838640c\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7;K8S_POD_INFRA_CONTAINER_ID=ed009013f5e0cdb4cecdff219276392e7c109f83af64a2a4dff02496cef2d18c;K8S_POD_UID=5e30a2f8-aa4d-4dea-8468-aa09e6100c58\\\" Path:\\\"\\\" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7/5e30a2f8-aa4d-4dea-8468-aa09e6100c58]: error waiting for pod: Get \\\"https://abc2cf8bc225b427fb144199fe91b720-31e182ef057dd652.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" podUID="5e30a2f8-aa4d-4dea-8468-aa09e6100c58" Oct 30 18:25:09.585927 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:09.585844 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-b8b654-6c7556f4cb-8prx6" podUID="9e68229e-2581-4c3b-8ab5-e1c867973ac1" containerName="payload-processing" containerID="cri-o://9475b7b587ac109c5bdace08cc733f99fdcfb779e966646b3d0bffae4b67280e" gracePeriod=25 Oct 30 18:25:09.625008 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:09.624953 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8b654-78678d94b7-nm799" podUID="705d36c4-b030-4767-86f8-351552d25d53" containerName="payload-pre-processing" containerID="cri-o://f4eec0b858ace8269d056683ad0642b4fb90cdce65a1d7a337aaea876f139d99" gracePeriod=25 Oct 30 18:25:10.191956 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:10.191927 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8b654-78678d94b7-nm799" Oct 30 18:25:10.194148 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:10.194120 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b8b654-6c7556f4cb-8prx6" Oct 30 18:25:10.302177 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:10.300788 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/705d36c4-b030-4767-86f8-351552d25d53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/705d36c4-b030-4767-86f8-351552d25d53-plugins-config-volume\") pod \"705d36c4-b030-4767-86f8-351552d25d53\" (UID: \"705d36c4-b030-4767-86f8-351552d25d53\") " Oct 30 18:25:10.302177 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:10.300885 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e68229e-2581-4c3b-8ab5-e1c867973ac1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e68229e-2581-4c3b-8ab5-e1c867973ac1-plugins-config-volume\") pod \"9e68229e-2581-4c3b-8ab5-e1c867973ac1\" (UID: \"9e68229e-2581-4c3b-8ab5-e1c867973ac1\") " Oct 30 18:25:10.302177 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:10.301049 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e68229e-2581-4c3b-8ab5-e1c867973ac1-kube-api-access-gzxkj\" (UniqueName: \"kubernetes.io/projected/9e68229e-2581-4c3b-8ab5-e1c867973ac1-kube-api-access-gzxkj\") pod \"9e68229e-2581-4c3b-8ab5-e1c867973ac1\" (UID: \"9e68229e-2581-4c3b-8ab5-e1c867973ac1\") " Oct 30 18:25:10.302177 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:10.301080 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/705d36c4-b030-4767-86f8-351552d25d53-kube-api-access-hdktv\" (UniqueName: \"kubernetes.io/projected/705d36c4-b030-4767-86f8-351552d25d53-kube-api-access-hdktv\") pod \"705d36c4-b030-4767-86f8-351552d25d53\" (UID: \"705d36c4-b030-4767-86f8-351552d25d53\") " Oct 30 18:25:10.302177 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:10.302119 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e68229e-2581-4c3b-8ab5-e1c867973ac1-plugins-config-volume" pod "9e68229e-2581-4c3b-8ab5-e1c867973ac1" (UID: "9e68229e-2581-4c3b-8ab5-e1c867973ac1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:10.302177 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:10.302123 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/705d36c4-b030-4767-86f8-351552d25d53-plugins-config-volume" pod "705d36c4-b030-4767-86f8-351552d25d53" (UID: "705d36c4-b030-4767-86f8-351552d25d53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:10.306852 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:10.306815 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e68229e-2581-4c3b-8ab5-e1c867973ac1-kube-api-access-gzxkj" pod "9e68229e-2581-4c3b-8ab5-e1c867973ac1" (UID: "9e68229e-2581-4c3b-8ab5-e1c867973ac1"). InnerVolumeSpecName "kube-api-access-gzxkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:10.311852 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:10.311819 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/705d36c4-b030-4767-86f8-351552d25d53-kube-api-access-hdktv" pod "705d36c4-b030-4767-86f8-351552d25d53" (UID: "705d36c4-b030-4767-86f8-351552d25d53"). InnerVolumeSpecName "kube-api-access-hdktv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:10.403229 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:10.403137 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/705d36c4-b030-4767-86f8-351552d25d53-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:10.403229 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:10.403177 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e68229e-2581-4c3b-8ab5-e1c867973ac1-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:10.403229 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:10.403192 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzxkj\" (UniqueName: \"kubernetes.io/projected/9e68229e-2581-4c3b-8ab5-e1c867973ac1-kube-api-access-gzxkj\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:10.403559 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:10.403235 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdktv\" (UniqueName: \"kubernetes.io/projected/705d36c4-b030-4767-86f8-351552d25d53-kube-api-access-hdktv\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:10.431345 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:10.428884 4455 generic.go:356] "Generic (PLEG): container finished" podID="705d36c4-b030-4767-86f8-351552d25d53" containerID="f4eec0b858ace8269d056683ad0642b4fb90cdce65a1d7a337aaea876f139d99" exitCode=0 Oct 30 18:25:10.431345 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:10.428936 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8b654-78678d94b7-nm799" Oct 30 18:25:10.431345 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:10.428966 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8b654-78678d94b7-nm799" event={"ID":"705d36c4-b030-4767-86f8-351552d25d53","Type":"ContainerDied","Data":"f4eec0b858ace8269d056683ad0642b4fb90cdce65a1d7a337aaea876f139d99"} Oct 30 18:25:10.431345 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:10.429040 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8b654-78678d94b7-nm799" event={"ID":"705d36c4-b030-4767-86f8-351552d25d53","Type":"ContainerDied","Data":"8605e69ddab8c399c4d66cae58e740b43803c14aa71d7f5c92985b27c00dde94"} Oct 30 18:25:10.431345 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:10.429071 4455 scope.go:122] "RemoveContainer" containerID="f4eec0b858ace8269d056683ad0642b4fb90cdce65a1d7a337aaea876f139d99" Oct 30 18:25:10.434601 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:10.434537 4455 generic.go:356] "Generic (PLEG): container finished" podID="9e68229e-2581-4c3b-8ab5-e1c867973ac1" containerID="9475b7b587ac109c5bdace08cc733f99fdcfb779e966646b3d0bffae4b67280e" exitCode=0 Oct 30 18:25:10.434601 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:10.434594 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b8b654-6c7556f4cb-8prx6" Oct 30 18:25:10.434752 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:10.434611 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b8b654-6c7556f4cb-8prx6" event={"ID":"9e68229e-2581-4c3b-8ab5-e1c867973ac1","Type":"ContainerDied","Data":"9475b7b587ac109c5bdace08cc733f99fdcfb779e966646b3d0bffae4b67280e"} Oct 30 18:25:10.434752 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:10.434658 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b8b654-6c7556f4cb-8prx6" event={"ID":"9e68229e-2581-4c3b-8ab5-e1c867973ac1","Type":"ContainerDied","Data":"6efe387f4bc4717cf9a20c1a48f61122d5b56f95c8f4eb0bf93494a5517fe817"} Oct 30 18:25:10.449827 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:10.449794 4455 scope.go:122] "RemoveContainer" containerID="f4eec0b858ace8269d056683ad0642b4fb90cdce65a1d7a337aaea876f139d99" Oct 30 18:25:10.450383 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:25:10.450210 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4eec0b858ace8269d056683ad0642b4fb90cdce65a1d7a337aaea876f139d99\": container with ID starting with f4eec0b858ace8269d056683ad0642b4fb90cdce65a1d7a337aaea876f139d99 not found: ID does not exist" containerID="f4eec0b858ace8269d056683ad0642b4fb90cdce65a1d7a337aaea876f139d99" Oct 30 18:25:10.450383 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:10.450254 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4eec0b858ace8269d056683ad0642b4fb90cdce65a1d7a337aaea876f139d99"} err="failed to get container status \"f4eec0b858ace8269d056683ad0642b4fb90cdce65a1d7a337aaea876f139d99\": rpc error: code = NotFound desc = could not find container \"f4eec0b858ace8269d056683ad0642b4fb90cdce65a1d7a337aaea876f139d99\": container with ID starting with f4eec0b858ace8269d056683ad0642b4fb90cdce65a1d7a337aaea876f139d99 not found: ID does not exist" Oct 30 18:25:10.450383 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:10.450277 4455 scope.go:122] "RemoveContainer" containerID="9475b7b587ac109c5bdace08cc733f99fdcfb779e966646b3d0bffae4b67280e" Oct 30 18:25:10.473097 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:10.470570 4455 scope.go:122] "RemoveContainer" containerID="9475b7b587ac109c5bdace08cc733f99fdcfb779e966646b3d0bffae4b67280e" Oct 30 18:25:10.473097 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:25:10.470970 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9475b7b587ac109c5bdace08cc733f99fdcfb779e966646b3d0bffae4b67280e\": container with ID starting with 9475b7b587ac109c5bdace08cc733f99fdcfb779e966646b3d0bffae4b67280e not found: ID does not exist" containerID="9475b7b587ac109c5bdace08cc733f99fdcfb779e966646b3d0bffae4b67280e" Oct 30 18:25:10.473097 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:10.471003 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9475b7b587ac109c5bdace08cc733f99fdcfb779e966646b3d0bffae4b67280e"} err="failed to get container status \"9475b7b587ac109c5bdace08cc733f99fdcfb779e966646b3d0bffae4b67280e\": rpc error: code = NotFound desc = could not find container \"9475b7b587ac109c5bdace08cc733f99fdcfb779e966646b3d0bffae4b67280e\": container with ID starting with 9475b7b587ac109c5bdace08cc733f99fdcfb779e966646b3d0bffae4b67280e not found: ID does not exist" Oct 30 18:25:10.566495 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:10.566443 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b8b654-6c7556f4cb-8prx6"] Oct 30 18:25:10.586413 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:10.586376 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b8b654-6c7556f4cb-8prx6"] Oct 30 18:25:10.615412 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:10.613210 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b8b654-78678d94b7-nm799"] Oct 30 18:25:10.634351 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:10.628602 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" Oct 30 18:25:10.634351 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:10.628900 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" Oct 30 18:25:10.638344 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:10.637957 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b8b654-78678d94b7-nm799"] Oct 30 18:25:11.665440 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:11.665402 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="705d36c4-b030-4767-86f8-351552d25d53" path="/var/lib/kubelet/pods/705d36c4-b030-4767-86f8-351552d25d53/volumes" Oct 30 18:25:11.666626 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:11.666604 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e68229e-2581-4c3b-8ab5-e1c867973ac1" path="/var/lib/kubelet/pods/9e68229e-2581-4c3b-8ab5-e1c867973ac1/volumes" Oct 30 18:25:12.814985 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:12.724815 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45"] Oct 30 18:25:14.475348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:14.474381 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" event={"ID":"bfbae34d-b472-4dd7-b862-dbb457726886","Type":"ContainerStarted","Data":"ce2154dd0c35efbe17b6e6c1f47b8f6c1001b87b6015026d5698ab73fb230959"} Oct 30 18:25:14.709361 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:14.705427 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9" podUID="484dd1d5-ee6a-4423-87c4-efbb4ba01767" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:14.867151 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:14.756145 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm" podUID="e6e19981-c4a3-403e-9265-e33461b1d35a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:14.893383 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:14.884804 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:15.832311 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:15.832266 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" event={"ID":"bfbae34d-b472-4dd7-b862-dbb457726886","Type":"ContainerStarted","Data":"0176d329f7d90e96028f076ff97a4e57f2d5bff8b91917144636d65cb5d6630b"} Oct 30 18:25:15.834186 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:15.834161 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" Oct 30 18:25:15.887467 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:15.887421 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:15.888626 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:15.888603 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:25:15.978351 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:15.956717 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" podStartSLOduration=25.231711184 podStartE2EDuration="25.956693404s" totalImagesPullingTime="724.98222ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:15.909411916 +0000 UTC m=+1968.397048953" watchObservedRunningTime="2026-10-30 18:25:15.956693404 +0000 UTC m=+1968.444330440" Oct 30 18:25:16.370349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:16.352004 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:25:16.370349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:16.352083 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" podUID="1d3471dc-aaf3-4060-a64d-2413e522e65a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:16.647860 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:16.645996 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl" Oct 30 18:25:16.647860 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:16.646450 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl" Oct 30 18:25:16.899403 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:16.876219 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:25:17.356348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:17.355591 4455 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:17.632445 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:17.628352 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9" Oct 30 18:25:17.661350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:17.636805 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9" Oct 30 18:25:17.902461 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:17.885870 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:25:18.124379 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:18.103804 4455 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:25:18.124379 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:18.103881 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" podUID="d4e8e077-c9f7-44c8-9dd5-d1c7a2403862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:18.610383 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:18.596909 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7ffac8-6d59c957dd-qplqj"] Oct 30 18:25:18.610383 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:18.596989 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7ffac8-fd6b99659-kp2l5"] Oct 30 18:25:18.915348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:18.897426 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:18.987350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:18.977050 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:25:19.146727 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:19.142168 4455 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 18:25:19.386530 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:19.385084 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl"] Oct 30 18:25:19.428006 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:19.427513 4455 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:25:19.428006 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:19.427580 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" podUID="70d7e6c8-8d0f-40f0-a533-58158542fb7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:19.800632 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:19.800579 4455 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:25:19.800805 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:19.800657 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" podUID="d7de539f-17e9-453c-af6b-e83fc7c91239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:20.080350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:20.079719 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:25:20.537848 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:20.537807 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl" event={"ID":"6a166721-a635-4df9-adfe-7887eb5e7b39","Type":"ContainerStarted","Data":"26990acb9af6b5fa533fdfab98834bd9a48762c22925799fc8396ef6feeccaee"} Oct 30 18:25:20.634687 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:20.634625 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:25:20.857141 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:25:20.857092 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46eee9bc_87c0_4421_92e7_547a494dbf3c.slice/crio-b7251e7e86241eaef63d771241c80ef18494023ae25fd7f7a3b5f2dae082b2c0 WatchSource:0}: Error finding container b7251e7e86241eaef63d771241c80ef18494023ae25fd7f7a3b5f2dae082b2c0: Status 404 returned error can't find the container with id b7251e7e86241eaef63d771241c80ef18494023ae25fd7f7a3b5f2dae082b2c0 Oct 30 18:25:20.871020 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:20.870987 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9"] Oct 30 18:25:21.519923 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:21.519877 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9" event={"ID":"46eee9bc-87c0-4421-92e7-547a494dbf3c","Type":"ContainerStarted","Data":"80818c409c04de2cd7a34df5600a5a2b271fb0fd8b404ae5d2969a81297e70c4"} Oct 30 18:25:21.520345 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:21.519957 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9" Oct 30 18:25:21.520345 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:21.519987 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9" event={"ID":"46eee9bc-87c0-4421-92e7-547a494dbf3c","Type":"ContainerStarted","Data":"b7251e7e86241eaef63d771241c80ef18494023ae25fd7f7a3b5f2dae082b2c0"} Oct 30 18:25:21.521642 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:21.521615 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl" event={"ID":"6a166721-a635-4df9-adfe-7887eb5e7b39","Type":"ContainerStarted","Data":"2971e893381552fe53bfd096f7337a5df321e7c50a127b113243cb274e212b23"} Oct 30 18:25:21.521752 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:21.521699 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl" Oct 30 18:25:21.565112 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:21.565056 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9" podStartSLOduration=23.565043523 podStartE2EDuration="23.565043523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:21.563810608 +0000 UTC m=+1974.051447653" watchObservedRunningTime="2026-10-30 18:25:21.565043523 +0000 UTC m=+1974.052680557" Oct 30 18:25:21.611697 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:21.611648 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl" podStartSLOduration=23.611635063 podStartE2EDuration="23.611635063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:21.60769525 +0000 UTC m=+1974.095332285" watchObservedRunningTime="2026-10-30 18:25:21.611635063 +0000 UTC m=+1974.099272097" Oct 30 18:25:22.438027 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:22.437988 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw"] Oct 30 18:25:22.438948 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:22.438929 4455 cpu_manager.go:414] "containerMap: removing container" podUID="9e68229e-2581-4c3b-8ab5-e1c867973ac1" containerName="payload-processing" Oct 30 18:25:22.439009 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:22.438950 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e68229e-2581-4c3b-8ab5-e1c867973ac1" containerName="payload-processing" Oct 30 18:25:22.439009 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:22.438979 4455 cpu_manager.go:414] "containerMap: removing container" podUID="705d36c4-b030-4767-86f8-351552d25d53" containerName="payload-pre-processing" Oct 30 18:25:22.439009 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:22.438985 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="705d36c4-b030-4767-86f8-351552d25d53" containerName="payload-pre-processing" Oct 30 18:25:22.439568 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:22.439111 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="705d36c4-b030-4767-86f8-351552d25d53" containerName="payload-pre-processing" Oct 30 18:25:22.439568 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:22.439128 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e68229e-2581-4c3b-8ab5-e1c867973ac1" containerName="payload-processing" Oct 30 18:25:22.443096 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:22.443073 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw" Oct 30 18:25:22.488362 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:22.479032 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-42fad7-serving-cert\"" Oct 30 18:25:22.488362 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:22.486263 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw"] Oct 30 18:25:22.581256 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:22.581217 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da2b820f-4f95-41a7-a438-32c71a5e3679-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw\" (UID: \"da2b820f-4f95-41a7-a438-32c71a5e3679\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw" Oct 30 18:25:22.581873 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:22.581845 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da2b820f-4f95-41a7-a438-32c71a5e3679-maas-api-tls\") pod \"maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw\" (UID: \"da2b820f-4f95-41a7-a438-32c71a5e3679\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw" Oct 30 18:25:22.582062 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:22.582044 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrx2t\" (UniqueName: \"kubernetes.io/projected/da2b820f-4f95-41a7-a438-32c71a5e3679-kube-api-access-wrx2t\") pod \"maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw\" (UID: \"da2b820f-4f95-41a7-a438-32c71a5e3679\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw" Oct 30 18:25:22.582160 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:22.582087 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da2b820f-4f95-41a7-a438-32c71a5e3679-metrics-certs\") pod \"maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw\" (UID: \"da2b820f-4f95-41a7-a438-32c71a5e3679\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw" Oct 30 18:25:22.626681 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:22.626648 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" Oct 30 18:25:22.627224 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:22.627207 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" Oct 30 18:25:22.684292 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:22.683577 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da2b820f-4f95-41a7-a438-32c71a5e3679-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw\" (UID: \"da2b820f-4f95-41a7-a438-32c71a5e3679\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw" Oct 30 18:25:22.684738 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:22.684706 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da2b820f-4f95-41a7-a438-32c71a5e3679-maas-api-tls\") pod \"maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw\" (UID: \"da2b820f-4f95-41a7-a438-32c71a5e3679\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw" Oct 30 18:25:22.685907 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:22.685161 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da2b820f-4f95-41a7-a438-32c71a5e3679-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw\" (UID: \"da2b820f-4f95-41a7-a438-32c71a5e3679\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw" Oct 30 18:25:22.686219 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:22.686203 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrx2t\" (UniqueName: \"kubernetes.io/projected/da2b820f-4f95-41a7-a438-32c71a5e3679-kube-api-access-wrx2t\") pod \"maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw\" (UID: \"da2b820f-4f95-41a7-a438-32c71a5e3679\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw" Oct 30 18:25:22.686718 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:22.686699 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da2b820f-4f95-41a7-a438-32c71a5e3679-metrics-certs\") pod \"maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw\" (UID: \"da2b820f-4f95-41a7-a438-32c71a5e3679\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw" Oct 30 18:25:22.688725 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:22.688658 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da2b820f-4f95-41a7-a438-32c71a5e3679-maas-api-tls\") pod \"maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw\" (UID: \"da2b820f-4f95-41a7-a438-32c71a5e3679\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw" Oct 30 18:25:22.690584 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:22.690542 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da2b820f-4f95-41a7-a438-32c71a5e3679-metrics-certs\") pod \"maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw\" (UID: \"da2b820f-4f95-41a7-a438-32c71a5e3679\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw" Oct 30 18:25:22.723417 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:22.723347 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrx2t\" (UniqueName: \"kubernetes.io/projected/da2b820f-4f95-41a7-a438-32c71a5e3679-kube-api-access-wrx2t\") pod \"maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw\" (UID: \"da2b820f-4f95-41a7-a438-32c71a5e3679\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw" Oct 30 18:25:22.765395 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:22.762613 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw" Oct 30 18:25:22.958784 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:22.958746 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7"] Oct 30 18:25:22.960227 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:25:22.960199 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e30a2f8_aa4d_4dea_8468_aa09e6100c58.slice/crio-9b74d5b69511d8c1b9812020f898641123b5df8e8eb70681422ff7c2cb3bb4a3 WatchSource:0}: Error finding container 9b74d5b69511d8c1b9812020f898641123b5df8e8eb70681422ff7c2cb3bb4a3: Status 404 returned error can't find the container with id 9b74d5b69511d8c1b9812020f898641123b5df8e8eb70681422ff7c2cb3bb4a3 Oct 30 18:25:23.123275 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:23.123250 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw"] Oct 30 18:25:23.123787 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:25:23.123755 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda2b820f_4f95_41a7_a438_32c71a5e3679.slice/crio-4523de5fc36a60071c94eaaf84035ed60918b355f3348c480e44de6cce368f8b WatchSource:0}: Error finding container 4523de5fc36a60071c94eaaf84035ed60918b355f3348c480e44de6cce368f8b: Status 404 returned error can't find the container with id 4523de5fc36a60071c94eaaf84035ed60918b355f3348c480e44de6cce368f8b Oct 30 18:25:23.541301 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:23.541252 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw" event={"ID":"da2b820f-4f95-41a7-a438-32c71a5e3679","Type":"ContainerStarted","Data":"9ade51f5e34a4b8a162c9a8646b71d79ba0b643050aaabcc3a87aeffbc87e3bc"} Oct 30 18:25:23.541555 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:23.541409 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw" event={"ID":"da2b820f-4f95-41a7-a438-32c71a5e3679","Type":"ContainerStarted","Data":"4523de5fc36a60071c94eaaf84035ed60918b355f3348c480e44de6cce368f8b"} Oct 30 18:25:23.541555 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:23.541548 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw" Oct 30 18:25:23.543288 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:23.543262 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" event={"ID":"5e30a2f8-aa4d-4dea-8468-aa09e6100c58","Type":"ContainerStarted","Data":"924369a89d9e84b6208d65e91061263346f386a85b508e3b633bfc1ad71f4afc"} Oct 30 18:25:23.543471 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:23.543309 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" event={"ID":"5e30a2f8-aa4d-4dea-8468-aa09e6100c58","Type":"ContainerStarted","Data":"9b74d5b69511d8c1b9812020f898641123b5df8e8eb70681422ff7c2cb3bb4a3"} Oct 30 18:25:23.543957 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:23.543935 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" Oct 30 18:25:23.598105 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:23.598061 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7ffac8-fd6b99659-kp2l5" podUID="73040eeb-fa91-44c3-9b4f-70fe18fcdaab" containerName="payload-pre-processing" containerID="cri-o://ace5220c593576a0bb7a962ee389a893ba15fa6bf67feebaafb227d3ed268bff" gracePeriod=25 Oct 30 18:25:23.598598 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:23.598045 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-7ffac8-6d59c957dd-qplqj" podUID="a754cbb1-3b1e-499d-b9e2-1a8445b4fd83" containerName="payload-processing" containerID="cri-o://376a9fe37ae8e11d1ca887485169eb801f1a87fadf2519c8cd1f649eedd23060" gracePeriod=25 Oct 30 18:25:23.639049 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:23.639001 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" podStartSLOduration=54.382116827 podStartE2EDuration="54.638986931s" totalImagesPullingTime="256.870104ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:23.637114787 +0000 UTC m=+1976.124751822" watchObservedRunningTime="2026-10-30 18:25:23.638986931 +0000 UTC m=+1976.126623964" Oct 30 18:25:23.639214 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:23.639124 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw" podStartSLOduration=1.314754399 podStartE2EDuration="1.639117376s" totalImagesPullingTime="324.362977ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:23.59162569 +0000 UTC m=+1976.079262725" watchObservedRunningTime="2026-10-30 18:25:23.639117376 +0000 UTC m=+1976.126754410" Oct 30 18:25:24.127114 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.127081 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7ffac8-6d59c957dd-qplqj" Oct 30 18:25:24.149834 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.149806 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" Oct 30 18:25:24.159095 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.159066 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7ffac8-fd6b99659-kp2l5" Oct 30 18:25:24.252421 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.252383 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a754cbb1-3b1e-499d-b9e2-1a8445b4fd83-kube-api-access-wvmc9\" (UniqueName: \"kubernetes.io/projected/a754cbb1-3b1e-499d-b9e2-1a8445b4fd83-kube-api-access-wvmc9\") pod \"a754cbb1-3b1e-499d-b9e2-1a8445b4fd83\" (UID: \"a754cbb1-3b1e-499d-b9e2-1a8445b4fd83\") " Oct 30 18:25:24.252658 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.252456 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73040eeb-fa91-44c3-9b4f-70fe18fcdaab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73040eeb-fa91-44c3-9b4f-70fe18fcdaab-plugins-config-volume\") pod \"73040eeb-fa91-44c3-9b4f-70fe18fcdaab\" (UID: \"73040eeb-fa91-44c3-9b4f-70fe18fcdaab\") " Oct 30 18:25:24.252658 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.252555 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a754cbb1-3b1e-499d-b9e2-1a8445b4fd83-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a754cbb1-3b1e-499d-b9e2-1a8445b4fd83-plugins-config-volume\") pod \"a754cbb1-3b1e-499d-b9e2-1a8445b4fd83\" (UID: \"a754cbb1-3b1e-499d-b9e2-1a8445b4fd83\") " Oct 30 18:25:24.253255 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.252892 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73040eeb-fa91-44c3-9b4f-70fe18fcdaab-kube-api-access-25wx2\" (UniqueName: \"kubernetes.io/projected/73040eeb-fa91-44c3-9b4f-70fe18fcdaab-kube-api-access-25wx2\") pod \"73040eeb-fa91-44c3-9b4f-70fe18fcdaab\" (UID: \"73040eeb-fa91-44c3-9b4f-70fe18fcdaab\") " Oct 30 18:25:24.253255 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.253241 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a754cbb1-3b1e-499d-b9e2-1a8445b4fd83-plugins-config-volume" pod "a754cbb1-3b1e-499d-b9e2-1a8445b4fd83" (UID: "a754cbb1-3b1e-499d-b9e2-1a8445b4fd83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:24.254013 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.253675 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a754cbb1-3b1e-499d-b9e2-1a8445b4fd83-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:24.254562 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.254538 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73040eeb-fa91-44c3-9b4f-70fe18fcdaab-plugins-config-volume" pod "73040eeb-fa91-44c3-9b4f-70fe18fcdaab" (UID: "73040eeb-fa91-44c3-9b4f-70fe18fcdaab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:24.256413 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.256389 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73040eeb-fa91-44c3-9b4f-70fe18fcdaab-kube-api-access-25wx2" pod "73040eeb-fa91-44c3-9b4f-70fe18fcdaab" (UID: "73040eeb-fa91-44c3-9b4f-70fe18fcdaab"). InnerVolumeSpecName "kube-api-access-25wx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:24.256683 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.256662 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a754cbb1-3b1e-499d-b9e2-1a8445b4fd83-kube-api-access-wvmc9" pod "a754cbb1-3b1e-499d-b9e2-1a8445b4fd83" (UID: "a754cbb1-3b1e-499d-b9e2-1a8445b4fd83"). InnerVolumeSpecName "kube-api-access-wvmc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:24.355487 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.355445 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvmc9\" (UniqueName: \"kubernetes.io/projected/a754cbb1-3b1e-499d-b9e2-1a8445b4fd83-kube-api-access-wvmc9\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:24.355487 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.355486 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73040eeb-fa91-44c3-9b4f-70fe18fcdaab-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:24.355679 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.355501 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25wx2\" (UniqueName: \"kubernetes.io/projected/73040eeb-fa91-44c3-9b4f-70fe18fcdaab-kube-api-access-25wx2\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:24.556019 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.555933 4455 generic.go:356] "Generic (PLEG): container finished" podID="73040eeb-fa91-44c3-9b4f-70fe18fcdaab" containerID="ace5220c593576a0bb7a962ee389a893ba15fa6bf67feebaafb227d3ed268bff" exitCode=0 Oct 30 18:25:24.556019 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.555964 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7ffac8-fd6b99659-kp2l5" event={"ID":"73040eeb-fa91-44c3-9b4f-70fe18fcdaab","Type":"ContainerDied","Data":"ace5220c593576a0bb7a962ee389a893ba15fa6bf67feebaafb227d3ed268bff"} Oct 30 18:25:24.556019 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.555998 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7ffac8-fd6b99659-kp2l5" Oct 30 18:25:24.556258 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.556036 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7ffac8-fd6b99659-kp2l5" event={"ID":"73040eeb-fa91-44c3-9b4f-70fe18fcdaab","Type":"ContainerDied","Data":"dad477254cdeb450910c1f9a82d920b19bc3039ae91cdf3fe2c87487663f97fe"} Oct 30 18:25:24.556258 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.556055 4455 scope.go:122] "RemoveContainer" containerID="ace5220c593576a0bb7a962ee389a893ba15fa6bf67feebaafb227d3ed268bff" Oct 30 18:25:24.558073 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.558046 4455 generic.go:356] "Generic (PLEG): container finished" podID="a754cbb1-3b1e-499d-b9e2-1a8445b4fd83" containerID="376a9fe37ae8e11d1ca887485169eb801f1a87fadf2519c8cd1f649eedd23060" exitCode=0 Oct 30 18:25:24.558205 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.558093 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7ffac8-6d59c957dd-qplqj" Oct 30 18:25:24.558205 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.558175 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7ffac8-6d59c957dd-qplqj" event={"ID":"a754cbb1-3b1e-499d-b9e2-1a8445b4fd83","Type":"ContainerDied","Data":"376a9fe37ae8e11d1ca887485169eb801f1a87fadf2519c8cd1f649eedd23060"} Oct 30 18:25:24.558425 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.558248 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7ffac8-6d59c957dd-qplqj" event={"ID":"a754cbb1-3b1e-499d-b9e2-1a8445b4fd83","Type":"ContainerDied","Data":"9a8cfa878a8f58f75957b5bcc0fdf64b3b70a34f008ba7a028f462c3bb307266"} Oct 30 18:25:24.573636 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.571148 4455 scope.go:122] "RemoveContainer" containerID="ace5220c593576a0bb7a962ee389a893ba15fa6bf67feebaafb227d3ed268bff" Oct 30 18:25:24.573636 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:25:24.571665 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ace5220c593576a0bb7a962ee389a893ba15fa6bf67feebaafb227d3ed268bff\": container with ID starting with ace5220c593576a0bb7a962ee389a893ba15fa6bf67feebaafb227d3ed268bff not found: ID does not exist" containerID="ace5220c593576a0bb7a962ee389a893ba15fa6bf67feebaafb227d3ed268bff" Oct 30 18:25:24.573636 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.571702 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ace5220c593576a0bb7a962ee389a893ba15fa6bf67feebaafb227d3ed268bff"} err="failed to get container status \"ace5220c593576a0bb7a962ee389a893ba15fa6bf67feebaafb227d3ed268bff\": rpc error: code = NotFound desc = could not find container \"ace5220c593576a0bb7a962ee389a893ba15fa6bf67feebaafb227d3ed268bff\": container with ID starting with ace5220c593576a0bb7a962ee389a893ba15fa6bf67feebaafb227d3ed268bff not found: ID does not exist" Oct 30 18:25:24.573636 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.571725 4455 scope.go:122] "RemoveContainer" containerID="376a9fe37ae8e11d1ca887485169eb801f1a87fadf2519c8cd1f649eedd23060" Oct 30 18:25:24.592582 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.592553 4455 scope.go:122] "RemoveContainer" containerID="376a9fe37ae8e11d1ca887485169eb801f1a87fadf2519c8cd1f649eedd23060" Oct 30 18:25:24.593055 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:25:24.592987 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"376a9fe37ae8e11d1ca887485169eb801f1a87fadf2519c8cd1f649eedd23060\": container with ID starting with 376a9fe37ae8e11d1ca887485169eb801f1a87fadf2519c8cd1f649eedd23060 not found: ID does not exist" containerID="376a9fe37ae8e11d1ca887485169eb801f1a87fadf2519c8cd1f649eedd23060" Oct 30 18:25:24.593055 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.593025 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"376a9fe37ae8e11d1ca887485169eb801f1a87fadf2519c8cd1f649eedd23060"} err="failed to get container status \"376a9fe37ae8e11d1ca887485169eb801f1a87fadf2519c8cd1f649eedd23060\": rpc error: code = NotFound desc = could not find container \"376a9fe37ae8e11d1ca887485169eb801f1a87fadf2519c8cd1f649eedd23060\": container with ID starting with 376a9fe37ae8e11d1ca887485169eb801f1a87fadf2519c8cd1f649eedd23060 not found: ID does not exist" Oct 30 18:25:24.636771 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.636741 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7ffac8-6d59c957dd-qplqj"] Oct 30 18:25:24.651509 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.651477 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7ffac8-6d59c957dd-qplqj"] Oct 30 18:25:24.689900 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.689868 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7ffac8-fd6b99659-kp2l5"] Oct 30 18:25:24.696842 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.696807 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7ffac8-fd6b99659-kp2l5"] Oct 30 18:25:24.861101 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.861029 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx"] Oct 30 18:25:24.861966 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.861947 4455 cpu_manager.go:414] "containerMap: removing container" podUID="73040eeb-fa91-44c3-9b4f-70fe18fcdaab" containerName="payload-pre-processing" Oct 30 18:25:24.862043 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.861969 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73040eeb-fa91-44c3-9b4f-70fe18fcdaab" containerName="payload-pre-processing" Oct 30 18:25:24.862089 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.862050 4455 cpu_manager.go:414] "containerMap: removing container" podUID="a754cbb1-3b1e-499d-b9e2-1a8445b4fd83" containerName="payload-processing" Oct 30 18:25:24.862089 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.862058 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a754cbb1-3b1e-499d-b9e2-1a8445b4fd83" containerName="payload-processing" Oct 30 18:25:24.862350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.862314 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="73040eeb-fa91-44c3-9b4f-70fe18fcdaab" containerName="payload-pre-processing" Oct 30 18:25:24.862410 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.862365 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="a754cbb1-3b1e-499d-b9e2-1a8445b4fd83" containerName="payload-processing" Oct 30 18:25:24.870632 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.868915 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" Oct 30 18:25:24.904563 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.904508 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx"] Oct 30 18:25:24.965786 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.965756 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-c2b103-openshift-default-dockercfg-2gsp2\"" Oct 30 18:25:24.965981 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.965958 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45acf61c-885f-4dc5-9ca3-d8293675107a-istio-token\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" Oct 30 18:25:24.966030 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.966005 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45acf61c-885f-4dc5-9ca3-d8293675107a-credential-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" Oct 30 18:25:24.966125 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.966096 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45acf61c-885f-4dc5-9ca3-d8293675107a-istio-data\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" Oct 30 18:25:24.966203 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.966190 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45acf61c-885f-4dc5-9ca3-d8293675107a-istio-envoy\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" Oct 30 18:25:24.966270 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.966257 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45acf61c-885f-4dc5-9ca3-d8293675107a-workload-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" Oct 30 18:25:24.966383 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.966366 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsfss\" (UniqueName: \"kubernetes.io/projected/45acf61c-885f-4dc5-9ca3-d8293675107a-kube-api-access-wsfss\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" Oct 30 18:25:24.966512 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.966440 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45acf61c-885f-4dc5-9ca3-d8293675107a-istio-podinfo\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" Oct 30 18:25:24.966512 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.966508 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45acf61c-885f-4dc5-9ca3-d8293675107a-workload-certs\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" Oct 30 18:25:24.966596 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:24.966558 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45acf61c-885f-4dc5-9ca3-d8293675107a-istiod-ca-cert\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" Oct 30 18:25:25.067713 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:25.067674 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsfss\" (UniqueName: \"kubernetes.io/projected/45acf61c-885f-4dc5-9ca3-d8293675107a-kube-api-access-wsfss\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" Oct 30 18:25:25.067880 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:25.067790 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45acf61c-885f-4dc5-9ca3-d8293675107a-istio-podinfo\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" Oct 30 18:25:25.067880 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:25.067824 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45acf61c-885f-4dc5-9ca3-d8293675107a-workload-certs\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" Oct 30 18:25:25.067986 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:25.067893 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45acf61c-885f-4dc5-9ca3-d8293675107a-istiod-ca-cert\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" Oct 30 18:25:25.068038 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:25.068001 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45acf61c-885f-4dc5-9ca3-d8293675107a-istio-token\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" Oct 30 18:25:25.068086 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:25.068044 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45acf61c-885f-4dc5-9ca3-d8293675107a-credential-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" Oct 30 18:25:25.068086 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:25.068078 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45acf61c-885f-4dc5-9ca3-d8293675107a-istio-data\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" Oct 30 18:25:25.068193 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:25.068107 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45acf61c-885f-4dc5-9ca3-d8293675107a-istio-envoy\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" Oct 30 18:25:25.068193 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:25.068176 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45acf61c-885f-4dc5-9ca3-d8293675107a-workload-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" Oct 30 18:25:25.069008 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:25.068443 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45acf61c-885f-4dc5-9ca3-d8293675107a-workload-certs\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" Oct 30 18:25:25.069008 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:25.068689 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45acf61c-885f-4dc5-9ca3-d8293675107a-credential-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" Oct 30 18:25:25.069008 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:25.068954 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45acf61c-885f-4dc5-9ca3-d8293675107a-istio-data\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" Oct 30 18:25:25.069008 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:25.068976 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45acf61c-885f-4dc5-9ca3-d8293675107a-istiod-ca-cert\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" Oct 30 18:25:25.069228 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:25.069093 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45acf61c-885f-4dc5-9ca3-d8293675107a-workload-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" Oct 30 18:25:25.070835 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:25.070811 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45acf61c-885f-4dc5-9ca3-d8293675107a-istio-podinfo\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" Oct 30 18:25:25.071219 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:25.071192 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45acf61c-885f-4dc5-9ca3-d8293675107a-istio-envoy\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" Oct 30 18:25:25.093472 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:25.093440 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsfss\" (UniqueName: \"kubernetes.io/projected/45acf61c-885f-4dc5-9ca3-d8293675107a-kube-api-access-wsfss\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" Oct 30 18:25:25.094298 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:25.094275 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45acf61c-885f-4dc5-9ca3-d8293675107a-istio-token\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" Oct 30 18:25:25.208932 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:25.208853 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" Oct 30 18:25:25.542391 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:25.542360 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx"] Oct 30 18:25:25.542862 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:25:25.542836 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45acf61c_885f_4dc5_9ca3_d8293675107a.slice/crio-a845733dc16e3b91a04c22b8ebc9d72fe59f4089c5fb7942b8162ca332a87d25 WatchSource:0}: Error finding container a845733dc16e3b91a04c22b8ebc9d72fe59f4089c5fb7942b8162ca332a87d25: Status 404 returned error can't find the container with id a845733dc16e3b91a04c22b8ebc9d72fe59f4089c5fb7942b8162ca332a87d25 Oct 30 18:25:25.545829 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:25.545793 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:25:25.545911 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:25.545869 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:25:25.545911 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:25.545903 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:25:25.571831 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:25.571795 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" event={"ID":"45acf61c-885f-4dc5-9ca3-d8293675107a","Type":"ContainerStarted","Data":"a845733dc16e3b91a04c22b8ebc9d72fe59f4089c5fb7942b8162ca332a87d25"} Oct 30 18:25:25.636795 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:25.636766 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73040eeb-fa91-44c3-9b4f-70fe18fcdaab" path="/var/lib/kubelet/pods/73040eeb-fa91-44c3-9b4f-70fe18fcdaab/volumes" Oct 30 18:25:25.637362 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:25.637340 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a754cbb1-3b1e-499d-b9e2-1a8445b4fd83" path="/var/lib/kubelet/pods/a754cbb1-3b1e-499d-b9e2-1a8445b4fd83/volumes" Oct 30 18:25:26.611571 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:26.611533 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" event={"ID":"45acf61c-885f-4dc5-9ca3-d8293675107a","Type":"ContainerStarted","Data":"a9f84fa6101c8e795a8d446b5021ccb549fd9517952f2308bbba9fe16fb641e3"} Oct 30 18:25:26.705589 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:26.705532 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" podStartSLOduration=2.705515224 podStartE2EDuration="2.705515224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:26.679021407 +0000 UTC m=+1979.166658456" watchObservedRunningTime="2026-10-30 18:25:26.705515224 +0000 UTC m=+1979.193152257" Oct 30 18:25:27.209353 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:27.209280 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" Oct 30 18:25:27.211442 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:27.211415 4455 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:25:27.211570 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:27.211466 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" podUID="45acf61c-885f-4dc5-9ca3-d8293675107a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 18:25:28.212353 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:28.209307 4455 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:25:28.212353 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:28.209394 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" podUID="45acf61c-885f-4dc5-9ca3-d8293675107a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 18:25:29.210041 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:29.210004 4455 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:25:29.210256 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:29.210060 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" podUID="45acf61c-885f-4dc5-9ca3-d8293675107a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 18:25:29.564933 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:29.564904 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw" Oct 30 18:25:30.211468 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:30.211151 4455 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:25:30.211468 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:30.211261 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" podUID="45acf61c-885f-4dc5-9ca3-d8293675107a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 18:25:30.415767 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:25:30.415733 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:25:30.603512 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:30.603474 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" Oct 30 18:25:31.209490 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:31.209450 4455 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:25:31.209692 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:31.209514 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" podUID="45acf61c-885f-4dc5-9ca3-d8293675107a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 18:25:31.526244 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:31.526200 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq"] Oct 30 18:25:31.526515 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:31.526484 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq" podUID="8d2b2365-0487-4dac-8482-eb697a969a09" containerName="maas-api" containerID="cri-o://886d3b49e1390f308b9bfaead68776952bb82e677b9bcb4b484cea00bd94d6d5" gracePeriod=30 Oct 30 18:25:31.658217 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:31.658187 4455 generic.go:356] "Generic (PLEG): container finished" podID="8d2b2365-0487-4dac-8482-eb697a969a09" containerID="886d3b49e1390f308b9bfaead68776952bb82e677b9bcb4b484cea00bd94d6d5" exitCode=0 Oct 30 18:25:31.658623 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:31.658226 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq" event={"ID":"8d2b2365-0487-4dac-8482-eb697a969a09","Type":"ContainerDied","Data":"886d3b49e1390f308b9bfaead68776952bb82e677b9bcb4b484cea00bd94d6d5"} Oct 30 18:25:31.914589 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:31.914558 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq" Oct 30 18:25:31.934919 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:31.934887 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d2b2365-0487-4dac-8482-eb697a969a09-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d2b2365-0487-4dac-8482-eb697a969a09-metrics-certs\") pod \"8d2b2365-0487-4dac-8482-eb697a969a09\" (UID: \"8d2b2365-0487-4dac-8482-eb697a969a09\") " Oct 30 18:25:31.935073 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:31.935042 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d2b2365-0487-4dac-8482-eb697a969a09-kube-api-access-qklvq\" (UniqueName: \"kubernetes.io/projected/8d2b2365-0487-4dac-8482-eb697a969a09-kube-api-access-qklvq\") pod \"8d2b2365-0487-4dac-8482-eb697a969a09\" (UID: \"8d2b2365-0487-4dac-8482-eb697a969a09\") " Oct 30 18:25:31.935193 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:31.935175 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d2b2365-0487-4dac-8482-eb697a969a09-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d2b2365-0487-4dac-8482-eb697a969a09-maas-api-tls\") pod \"8d2b2365-0487-4dac-8482-eb697a969a09\" (UID: \"8d2b2365-0487-4dac-8482-eb697a969a09\") " Oct 30 18:25:31.935254 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:31.935234 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d2b2365-0487-4dac-8482-eb697a969a09-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d2b2365-0487-4dac-8482-eb697a969a09-openshift-service-ca\") pod \"8d2b2365-0487-4dac-8482-eb697a969a09\" (UID: \"8d2b2365-0487-4dac-8482-eb697a969a09\") " Oct 30 18:25:31.935639 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:31.935607 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d2b2365-0487-4dac-8482-eb697a969a09-openshift-service-ca" pod "8d2b2365-0487-4dac-8482-eb697a969a09" (UID: "8d2b2365-0487-4dac-8482-eb697a969a09"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:31.935973 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:31.935923 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d2b2365-0487-4dac-8482-eb697a969a09-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:31.937452 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:31.937425 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d2b2365-0487-4dac-8482-eb697a969a09-maas-api-tls" pod "8d2b2365-0487-4dac-8482-eb697a969a09" (UID: "8d2b2365-0487-4dac-8482-eb697a969a09"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:31.937739 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:31.937717 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d2b2365-0487-4dac-8482-eb697a969a09-metrics-certs" pod "8d2b2365-0487-4dac-8482-eb697a969a09" (UID: "8d2b2365-0487-4dac-8482-eb697a969a09"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:31.937810 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:31.937734 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d2b2365-0487-4dac-8482-eb697a969a09-kube-api-access-qklvq" pod "8d2b2365-0487-4dac-8482-eb697a969a09" (UID: "8d2b2365-0487-4dac-8482-eb697a969a09"). InnerVolumeSpecName "kube-api-access-qklvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:32.037353 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:32.037288 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d2b2365-0487-4dac-8482-eb697a969a09-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:32.037353 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:32.037337 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qklvq\" (UniqueName: \"kubernetes.io/projected/8d2b2365-0487-4dac-8482-eb697a969a09-kube-api-access-qklvq\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:32.037353 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:32.037348 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d2b2365-0487-4dac-8482-eb697a969a09-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:32.273393 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:32.264666 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" Oct 30 18:25:32.273393 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:32.266310 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" Oct 30 18:25:32.284347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:32.275669 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" Oct 30 18:25:32.533660 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:32.533574 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl" Oct 30 18:25:32.533660 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:32.533640 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9" Oct 30 18:25:32.684356 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:32.677282 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq" Oct 30 18:25:32.684356 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:32.681939 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq" event={"ID":"8d2b2365-0487-4dac-8482-eb697a969a09","Type":"ContainerDied","Data":"b366109fa502396fe06b1686fc05f0a943680745c2a65f107047f2424d310e55"} Oct 30 18:25:32.684356 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:32.682005 4455 scope.go:122] "RemoveContainer" containerID="886d3b49e1390f308b9bfaead68776952bb82e677b9bcb4b484cea00bd94d6d5" Oct 30 18:25:32.762379 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:32.762283 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq"] Oct 30 18:25:32.767393 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:32.767364 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8b654-74d946d7ff-fwvmq"] Oct 30 18:25:33.636707 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:33.636671 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d2b2365-0487-4dac-8482-eb697a969a09" path="/var/lib/kubelet/pods/8d2b2365-0487-4dac-8482-eb697a969a09/volumes" Oct 30 18:25:35.010369 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.010314 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg"] Oct 30 18:25:35.011174 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.011157 4455 cpu_manager.go:414] "containerMap: removing container" podUID="8d2b2365-0487-4dac-8482-eb697a969a09" containerName="maas-api" Oct 30 18:25:35.011226 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.011175 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d2b2365-0487-4dac-8482-eb697a969a09" containerName="maas-api" Oct 30 18:25:35.011293 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.011283 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d2b2365-0487-4dac-8482-eb697a969a09" containerName="maas-api" Oct 30 18:25:35.016195 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.016171 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" Oct 30 18:25:35.058009 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.057975 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg"] Oct 30 18:25:35.110235 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.110200 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3c22942-b988-4208-ba9f-5b8becffa013-istio-token\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" Oct 30 18:25:35.110429 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.110407 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c22942-b988-4208-ba9f-5b8becffa013-credential-socket\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" Oct 30 18:25:35.110495 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.110457 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3c22942-b988-4208-ba9f-5b8becffa013-istio-envoy\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" Oct 30 18:25:35.110610 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.110585 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3c22942-b988-4208-ba9f-5b8becffa013-istiod-ca-cert\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" Oct 30 18:25:35.110739 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.110715 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3c22942-b988-4208-ba9f-5b8becffa013-istio-data\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" Oct 30 18:25:35.110785 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.110753 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3c22942-b988-4208-ba9f-5b8becffa013-istio-podinfo\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" Oct 30 18:25:35.110836 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.110787 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3c22942-b988-4208-ba9f-5b8becffa013-workload-certs\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" Oct 30 18:25:35.110897 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.110845 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xxrf\" (UniqueName: \"kubernetes.io/projected/a3c22942-b988-4208-ba9f-5b8becffa013-kube-api-access-5xxrf\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" Oct 30 18:25:35.110950 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.110917 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c22942-b988-4208-ba9f-5b8becffa013-workload-socket\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" Oct 30 18:25:35.116113 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.116088 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-7ffac8-openshift-default-dockercfg-8kwr2\"" Oct 30 18:25:35.212475 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.212431 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3c22942-b988-4208-ba9f-5b8becffa013-istiod-ca-cert\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" Oct 30 18:25:35.212668 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.212542 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3c22942-b988-4208-ba9f-5b8becffa013-istio-data\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" Oct 30 18:25:35.212668 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.212571 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3c22942-b988-4208-ba9f-5b8becffa013-istio-podinfo\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" Oct 30 18:25:35.212668 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.212609 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3c22942-b988-4208-ba9f-5b8becffa013-workload-certs\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" Oct 30 18:25:35.212668 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.212659 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xxrf\" (UniqueName: \"kubernetes.io/projected/a3c22942-b988-4208-ba9f-5b8becffa013-kube-api-access-5xxrf\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" Oct 30 18:25:35.212906 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.212704 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c22942-b988-4208-ba9f-5b8becffa013-workload-socket\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" Oct 30 18:25:35.212906 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.212827 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3c22942-b988-4208-ba9f-5b8becffa013-istio-token\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" Oct 30 18:25:35.213019 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.212971 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c22942-b988-4208-ba9f-5b8becffa013-credential-socket\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" Oct 30 18:25:35.213604 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.213019 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3c22942-b988-4208-ba9f-5b8becffa013-istio-envoy\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" Oct 30 18:25:35.213604 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.213056 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3c22942-b988-4208-ba9f-5b8becffa013-workload-certs\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" Oct 30 18:25:35.213604 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.213124 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c22942-b988-4208-ba9f-5b8becffa013-workload-socket\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" Oct 30 18:25:35.213604 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.213234 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3c22942-b988-4208-ba9f-5b8becffa013-istiod-ca-cert\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" Oct 30 18:25:35.213604 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.213403 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c22942-b988-4208-ba9f-5b8becffa013-credential-socket\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" Oct 30 18:25:35.213604 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.213584 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3c22942-b988-4208-ba9f-5b8becffa013-istio-data\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" Oct 30 18:25:35.216341 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.216295 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3c22942-b988-4208-ba9f-5b8becffa013-istio-envoy\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" Oct 30 18:25:35.217124 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.217102 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3c22942-b988-4208-ba9f-5b8becffa013-istio-podinfo\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" Oct 30 18:25:35.237521 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.237491 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3c22942-b988-4208-ba9f-5b8becffa013-istio-token\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" Oct 30 18:25:35.237701 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.237681 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xxrf\" (UniqueName: \"kubernetes.io/projected/a3c22942-b988-4208-ba9f-5b8becffa013-kube-api-access-5xxrf\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" Oct 30 18:25:35.346794 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.346708 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" Oct 30 18:25:35.679713 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:25:35.679679 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c22942_b988_4208_ba9f_5b8becffa013.slice/crio-8b1ac3c2403681c325b12f7620f895af847066925be36864a9a8ceca6e92e2bb WatchSource:0}: Error finding container 8b1ac3c2403681c325b12f7620f895af847066925be36864a9a8ceca6e92e2bb: Status 404 returned error can't find the container with id 8b1ac3c2403681c325b12f7620f895af847066925be36864a9a8ceca6e92e2bb Oct 30 18:25:35.682107 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.682076 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg"] Oct 30 18:25:35.682881 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.682844 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:25:35.683000 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.682929 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:25:35.683000 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.682975 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:25:35.708018 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:35.707982 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" event={"ID":"a3c22942-b988-4208-ba9f-5b8becffa013","Type":"ContainerStarted","Data":"8b1ac3c2403681c325b12f7620f895af847066925be36864a9a8ceca6e92e2bb"} Oct 30 18:25:36.718459 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:36.718420 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" event={"ID":"a3c22942-b988-4208-ba9f-5b8becffa013","Type":"ContainerStarted","Data":"4c6dd46092dbffdeb7d9d59f59a2590d70169a1bab86aa8ad79a1ccb6020cbc7"} Oct 30 18:25:37.347604 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:37.347572 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" Oct 30 18:25:37.349734 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:37.349690 4455 patch_prober.go:29] interesting pod/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 18:25:37.349884 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:37.349772 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" podUID="a3c22942-b988-4208-ba9f-5b8becffa013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 18:25:38.347684 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:38.347650 4455 patch_prober.go:29] interesting pod/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 18:25:38.348174 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:38.347702 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" podUID="a3c22942-b988-4208-ba9f-5b8becffa013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 18:25:39.348063 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.348025 4455 patch_prober.go:29] interesting pod/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 18:25:39.348588 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.348081 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" podUID="a3c22942-b988-4208-ba9f-5b8becffa013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 18:25:39.362743 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.362673 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" podStartSLOduration=5.362658083 podStartE2EDuration="5.362658083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:36.804509238 +0000 UTC m=+1989.292146273" watchObservedRunningTime="2026-10-30 18:25:39.362658083 +0000 UTC m=+1991.850295117" Oct 30 18:25:39.363064 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.363048 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj"] Oct 30 18:25:39.387344 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.370613 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" Oct 30 18:25:39.404749 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.404718 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-b8b654-openshift-default-dockercfg-lrm47\"" Oct 30 18:25:39.417635 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.417600 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj"] Oct 30 18:25:39.517840 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.517803 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcw7q\" (UniqueName: \"kubernetes.io/projected/ad790c3e-7a8f-462a-835c-870bb8f73002-kube-api-access-zcw7q\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" Oct 30 18:25:39.517840 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.517843 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad790c3e-7a8f-462a-835c-870bb8f73002-credential-socket\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" Oct 30 18:25:39.518104 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.517875 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad790c3e-7a8f-462a-835c-870bb8f73002-istio-envoy\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" Oct 30 18:25:39.518104 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.518063 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad790c3e-7a8f-462a-835c-870bb8f73002-workload-socket\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" Oct 30 18:25:39.518205 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.518105 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad790c3e-7a8f-462a-835c-870bb8f73002-istiod-ca-cert\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" Oct 30 18:25:39.518265 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.518246 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad790c3e-7a8f-462a-835c-870bb8f73002-istio-data\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" Oct 30 18:25:39.518311 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.518296 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad790c3e-7a8f-462a-835c-870bb8f73002-istio-podinfo\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" Oct 30 18:25:39.518426 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.518396 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad790c3e-7a8f-462a-835c-870bb8f73002-istio-token\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" Oct 30 18:25:39.518548 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.518537 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad790c3e-7a8f-462a-835c-870bb8f73002-workload-certs\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" Oct 30 18:25:39.619735 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.619654 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad790c3e-7a8f-462a-835c-870bb8f73002-workload-certs\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" Oct 30 18:25:39.619735 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.619731 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcw7q\" (UniqueName: \"kubernetes.io/projected/ad790c3e-7a8f-462a-835c-870bb8f73002-kube-api-access-zcw7q\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" Oct 30 18:25:39.620056 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.619754 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad790c3e-7a8f-462a-835c-870bb8f73002-credential-socket\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" Oct 30 18:25:39.620056 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.619785 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad790c3e-7a8f-462a-835c-870bb8f73002-istio-envoy\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" Oct 30 18:25:39.620056 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.619841 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad790c3e-7a8f-462a-835c-870bb8f73002-workload-socket\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" Oct 30 18:25:39.620056 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.619864 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad790c3e-7a8f-462a-835c-870bb8f73002-istiod-ca-cert\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" Oct 30 18:25:39.620056 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.619960 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad790c3e-7a8f-462a-835c-870bb8f73002-istio-data\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" Oct 30 18:25:39.620056 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.620012 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad790c3e-7a8f-462a-835c-870bb8f73002-istio-podinfo\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" Oct 30 18:25:39.620401 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.620084 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad790c3e-7a8f-462a-835c-870bb8f73002-istio-token\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" Oct 30 18:25:39.620401 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.620093 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad790c3e-7a8f-462a-835c-870bb8f73002-workload-certs\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" Oct 30 18:25:39.620401 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.620365 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad790c3e-7a8f-462a-835c-870bb8f73002-workload-socket\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" Oct 30 18:25:39.620581 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.620557 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad790c3e-7a8f-462a-835c-870bb8f73002-istio-data\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" Oct 30 18:25:39.620910 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.620826 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad790c3e-7a8f-462a-835c-870bb8f73002-credential-socket\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" Oct 30 18:25:39.621110 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.620891 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad790c3e-7a8f-462a-835c-870bb8f73002-istiod-ca-cert\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" Oct 30 18:25:39.623418 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.623391 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad790c3e-7a8f-462a-835c-870bb8f73002-istio-envoy\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" Oct 30 18:25:39.623696 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.623673 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad790c3e-7a8f-462a-835c-870bb8f73002-istio-podinfo\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" Oct 30 18:25:39.645930 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.645906 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcw7q\" (UniqueName: \"kubernetes.io/projected/ad790c3e-7a8f-462a-835c-870bb8f73002-kube-api-access-zcw7q\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" Oct 30 18:25:39.646109 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.645911 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad790c3e-7a8f-462a-835c-870bb8f73002-istio-token\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" Oct 30 18:25:39.721268 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.721230 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" Oct 30 18:25:39.939419 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.939315 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk"] Oct 30 18:25:39.948282 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.948257 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" Oct 30 18:25:39.951438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.951369 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb055040-07c9-46eb-8649-dd1db076ee59-istio-data\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" Oct 30 18:25:39.951655 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.951632 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb055040-07c9-46eb-8649-dd1db076ee59-istio-token\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" Oct 30 18:25:39.951827 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.951807 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb055040-07c9-46eb-8649-dd1db076ee59-istio-podinfo\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" Oct 30 18:25:39.951985 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.951970 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb055040-07c9-46eb-8649-dd1db076ee59-istiod-ca-cert\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" Oct 30 18:25:39.952100 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.952082 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx6vl\" (UniqueName: \"kubernetes.io/projected/cb055040-07c9-46eb-8649-dd1db076ee59-kube-api-access-cx6vl\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" Oct 30 18:25:39.952220 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.952205 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb055040-07c9-46eb-8649-dd1db076ee59-workload-certs\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" Oct 30 18:25:39.952365 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.952348 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb055040-07c9-46eb-8649-dd1db076ee59-credential-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" Oct 30 18:25:39.952532 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.952515 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb055040-07c9-46eb-8649-dd1db076ee59-istio-envoy\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" Oct 30 18:25:39.952833 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:39.952797 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb055040-07c9-46eb-8649-dd1db076ee59-workload-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" Oct 30 18:25:40.038394 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.032883 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk"] Oct 30 18:25:40.038394 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.032966 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj"] Oct 30 18:25:40.076350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.059737 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb055040-07c9-46eb-8649-dd1db076ee59-workload-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" Oct 30 18:25:40.076350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.059885 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb055040-07c9-46eb-8649-dd1db076ee59-istio-data\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" Oct 30 18:25:40.076350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.059924 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb055040-07c9-46eb-8649-dd1db076ee59-istio-token\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" Oct 30 18:25:40.076350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.059999 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb055040-07c9-46eb-8649-dd1db076ee59-istio-podinfo\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" Oct 30 18:25:40.076350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.060050 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb055040-07c9-46eb-8649-dd1db076ee59-istiod-ca-cert\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" Oct 30 18:25:40.076350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.060081 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx6vl\" (UniqueName: \"kubernetes.io/projected/cb055040-07c9-46eb-8649-dd1db076ee59-kube-api-access-cx6vl\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" Oct 30 18:25:40.076350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.060111 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb055040-07c9-46eb-8649-dd1db076ee59-workload-certs\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" Oct 30 18:25:40.076350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.060157 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb055040-07c9-46eb-8649-dd1db076ee59-credential-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" Oct 30 18:25:40.076350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.060232 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb055040-07c9-46eb-8649-dd1db076ee59-istio-envoy\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" Oct 30 18:25:40.076350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.062366 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb055040-07c9-46eb-8649-dd1db076ee59-istiod-ca-cert\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" Oct 30 18:25:40.076350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.062642 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb055040-07c9-46eb-8649-dd1db076ee59-workload-certs\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" Oct 30 18:25:40.076350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.063145 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb055040-07c9-46eb-8649-dd1db076ee59-workload-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" Oct 30 18:25:40.076350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.063480 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb055040-07c9-46eb-8649-dd1db076ee59-credential-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" Oct 30 18:25:40.076350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.063740 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb055040-07c9-46eb-8649-dd1db076ee59-istio-data\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" Oct 30 18:25:40.101705 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.101188 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb055040-07c9-46eb-8649-dd1db076ee59-istio-envoy\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" Oct 30 18:25:40.104800 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.104738 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb055040-07c9-46eb-8649-dd1db076ee59-istio-token\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" Oct 30 18:25:40.105514 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.105487 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb055040-07c9-46eb-8649-dd1db076ee59-istio-podinfo\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" Oct 30 18:25:40.106772 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.106724 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx6vl\" (UniqueName: \"kubernetes.io/projected/cb055040-07c9-46eb-8649-dd1db076ee59-kube-api-access-cx6vl\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" Oct 30 18:25:40.352108 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.350618 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" Oct 30 18:25:40.400373 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.397514 4455 patch_prober.go:29] interesting pod/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 18:25:40.400373 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.397575 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" podUID="a3c22942-b988-4208-ba9f-5b8becffa013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 18:25:40.426252 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.415127 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj"] Oct 30 18:25:40.455472 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.455426 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:25:40.455667 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.455521 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:25:40.455667 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.455552 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:25:40.782366 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.774597 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg"] Oct 30 18:25:40.782366 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.779571 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" Oct 30 18:25:40.815923 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.806676 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg"] Oct 30 18:25:40.820613 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.820039 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" event={"ID":"ad790c3e-7a8f-462a-835c-870bb8f73002","Type":"ContainerStarted","Data":"44de37a7d527887cc6b6fb82249dd05aa09477668c5a67ffa5dbf65ce54152ab"} Oct 30 18:25:40.820613 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.820108 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" event={"ID":"ad790c3e-7a8f-462a-835c-870bb8f73002","Type":"ContainerStarted","Data":"a61df0114aa9aa1d00b6b3caef8e8ac10a0d0d600731e6a1a435e2eac9f316a2"} Oct 30 18:25:40.820613 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.820261 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" podUID="ad790c3e-7a8f-462a-835c-870bb8f73002" containerName="istio-proxy" containerID="cri-o://44de37a7d527887cc6b6fb82249dd05aa09477668c5a67ffa5dbf65ce54152ab" gracePeriod=30 Oct 30 18:25:40.822154 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.822105 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-77db82-openshift-default-dockercfg-btt49\"" Oct 30 18:25:40.863854 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.863814 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkscg\" (UniqueName: \"kubernetes.io/projected/7894da9d-a98c-45ac-9968-bfe4e9747c49-kube-api-access-zkscg\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" Oct 30 18:25:40.864052 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.863893 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7894da9d-a98c-45ac-9968-bfe4e9747c49-workload-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" Oct 30 18:25:40.864052 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.863940 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7894da9d-a98c-45ac-9968-bfe4e9747c49-istiod-ca-cert\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" Oct 30 18:25:40.864052 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.863981 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7894da9d-a98c-45ac-9968-bfe4e9747c49-istio-data\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" Oct 30 18:25:40.864052 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.864012 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7894da9d-a98c-45ac-9968-bfe4e9747c49-istio-podinfo\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" Oct 30 18:25:40.864052 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.864040 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7894da9d-a98c-45ac-9968-bfe4e9747c49-credential-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" Oct 30 18:25:40.864246 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.864165 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7894da9d-a98c-45ac-9968-bfe4e9747c49-workload-certs\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" Oct 30 18:25:40.864315 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.864272 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7894da9d-a98c-45ac-9968-bfe4e9747c49-istio-envoy\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" Oct 30 18:25:40.864547 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.864417 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7894da9d-a98c-45ac-9968-bfe4e9747c49-istio-token\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" Oct 30 18:25:40.991355 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.984536 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkscg\" (UniqueName: \"kubernetes.io/projected/7894da9d-a98c-45ac-9968-bfe4e9747c49-kube-api-access-zkscg\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" Oct 30 18:25:40.991355 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.984627 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7894da9d-a98c-45ac-9968-bfe4e9747c49-workload-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" Oct 30 18:25:40.991355 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.984675 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7894da9d-a98c-45ac-9968-bfe4e9747c49-istiod-ca-cert\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" Oct 30 18:25:40.991355 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.984720 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7894da9d-a98c-45ac-9968-bfe4e9747c49-istio-data\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" Oct 30 18:25:40.991355 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.984754 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7894da9d-a98c-45ac-9968-bfe4e9747c49-istio-podinfo\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" Oct 30 18:25:40.991355 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.984783 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7894da9d-a98c-45ac-9968-bfe4e9747c49-credential-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" Oct 30 18:25:40.991355 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.984901 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7894da9d-a98c-45ac-9968-bfe4e9747c49-workload-certs\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" Oct 30 18:25:40.991355 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.985001 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7894da9d-a98c-45ac-9968-bfe4e9747c49-istio-envoy\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" Oct 30 18:25:40.991355 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.985122 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7894da9d-a98c-45ac-9968-bfe4e9747c49-istio-token\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" Oct 30 18:25:40.991355 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.985933 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7894da9d-a98c-45ac-9968-bfe4e9747c49-workload-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" Oct 30 18:25:40.991355 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.986639 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7894da9d-a98c-45ac-9968-bfe4e9747c49-istiod-ca-cert\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" Oct 30 18:25:40.991355 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:40.986906 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7894da9d-a98c-45ac-9968-bfe4e9747c49-istio-data\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" Oct 30 18:25:41.001349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:41.000216 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7894da9d-a98c-45ac-9968-bfe4e9747c49-workload-certs\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" Oct 30 18:25:41.001349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:41.000242 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7894da9d-a98c-45ac-9968-bfe4e9747c49-credential-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" Oct 30 18:25:41.002892 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:41.002862 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7894da9d-a98c-45ac-9968-bfe4e9747c49-istio-podinfo\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" Oct 30 18:25:41.005166 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:41.005139 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7894da9d-a98c-45ac-9968-bfe4e9747c49-istio-envoy\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" Oct 30 18:25:41.025627 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:41.025570 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" podStartSLOduration=2.02555108 podStartE2EDuration="2.02555108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:40.91076637 +0000 UTC m=+1993.398403409" watchObservedRunningTime="2026-10-30 18:25:41.02555108 +0000 UTC m=+1993.513188115" Oct 30 18:25:41.037806 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:41.032862 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7894da9d-a98c-45ac-9968-bfe4e9747c49-istio-token\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" Oct 30 18:25:41.037806 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:41.033939 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkscg\" (UniqueName: \"kubernetes.io/projected/7894da9d-a98c-45ac-9968-bfe4e9747c49-kube-api-access-zkscg\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" Oct 30 18:25:41.060957 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:41.060902 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:25:41.061088 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:41.061055 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:25:41.061160 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:41.061139 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:25:41.121849 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:41.121818 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" Oct 30 18:25:41.125492 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:41.125239 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk"] Oct 30 18:25:41.229400 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:41.228694 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69"] Oct 30 18:25:41.229400 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:41.228991 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69" podUID="80b01b7a-2c9f-41b5-ba92-9ade95a89cc8" containerName="maas-api" containerID="cri-o://c18a87dab12929ae8d261322054c87f35bc7585013af55fc369120b39423d8c4" gracePeriod=30 Oct 30 18:25:41.356772 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:41.347734 4455 patch_prober.go:29] interesting pod/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 18:25:41.356772 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:41.347796 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" podUID="a3c22942-b988-4208-ba9f-5b8becffa013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 18:25:42.007604 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:42.007573 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg"] Oct 30 18:25:42.249779 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:42.237103 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:25:42.249779 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:42.237204 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:25:42.249779 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:42.237252 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:25:42.249779 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:42.237738 4455 generic.go:356] "Generic (PLEG): container finished" podID="80b01b7a-2c9f-41b5-ba92-9ade95a89cc8" containerID="c18a87dab12929ae8d261322054c87f35bc7585013af55fc369120b39423d8c4" exitCode=0 Oct 30 18:25:42.249779 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:42.237826 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69" event={"ID":"80b01b7a-2c9f-41b5-ba92-9ade95a89cc8","Type":"ContainerDied","Data":"c18a87dab12929ae8d261322054c87f35bc7585013af55fc369120b39423d8c4"} Oct 30 18:25:42.249779 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:42.237881 4455 scope.go:122] "RemoveContainer" containerID="8af70fe921a366f858d2bfde039bf60ee64e59652dee0cafdee698b46e80551f" Oct 30 18:25:42.251385 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:42.251233 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" event={"ID":"cb055040-07c9-46eb-8649-dd1db076ee59","Type":"ContainerStarted","Data":"0d334a0273c56ccdca0232e0f97b01f554b5972b51a90a1003fecf9847490c6d"} Oct 30 18:25:42.251385 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:42.251307 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" event={"ID":"cb055040-07c9-46eb-8649-dd1db076ee59","Type":"ContainerStarted","Data":"42a211f08fe3e0f8cd779102c3854e1178663069fc5b776bc3132144c96eb265"} Oct 30 18:25:42.354073 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:42.351190 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" Oct 30 18:25:42.354073 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:42.352756 4455 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 18:25:42.354073 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:42.353051 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" podUID="cb055040-07c9-46eb-8649-dd1db076ee59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 18:25:42.356721 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:42.356392 4455 patch_prober.go:29] interesting pod/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 18:25:42.356721 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:42.356448 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" podUID="a3c22942-b988-4208-ba9f-5b8becffa013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 18:25:42.416353 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:42.415348 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" podStartSLOduration=3.415314903 podStartE2EDuration="3.415314903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:42.412660894 +0000 UTC m=+1994.900297928" watchObservedRunningTime="2026-10-30 18:25:42.415314903 +0000 UTC m=+1994.902951938" Oct 30 18:25:42.490633 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:42.489507 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69" Oct 30 18:25:42.556269 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:42.555797 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80b01b7a-2c9f-41b5-ba92-9ade95a89cc8-kube-api-access-qrrw4\" (UniqueName: \"kubernetes.io/projected/80b01b7a-2c9f-41b5-ba92-9ade95a89cc8-kube-api-access-qrrw4\") pod \"80b01b7a-2c9f-41b5-ba92-9ade95a89cc8\" (UID: \"80b01b7a-2c9f-41b5-ba92-9ade95a89cc8\") " Oct 30 18:25:42.556269 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:42.555883 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80b01b7a-2c9f-41b5-ba92-9ade95a89cc8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80b01b7a-2c9f-41b5-ba92-9ade95a89cc8-maas-api-tls\") pod \"80b01b7a-2c9f-41b5-ba92-9ade95a89cc8\" (UID: \"80b01b7a-2c9f-41b5-ba92-9ade95a89cc8\") " Oct 30 18:25:42.556269 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:42.555948 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80b01b7a-2c9f-41b5-ba92-9ade95a89cc8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80b01b7a-2c9f-41b5-ba92-9ade95a89cc8-openshift-service-ca\") pod \"80b01b7a-2c9f-41b5-ba92-9ade95a89cc8\" (UID: \"80b01b7a-2c9f-41b5-ba92-9ade95a89cc8\") " Oct 30 18:25:42.556269 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:42.556216 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80b01b7a-2c9f-41b5-ba92-9ade95a89cc8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b01b7a-2c9f-41b5-ba92-9ade95a89cc8-metrics-certs\") pod \"80b01b7a-2c9f-41b5-ba92-9ade95a89cc8\" (UID: \"80b01b7a-2c9f-41b5-ba92-9ade95a89cc8\") " Oct 30 18:25:42.560293 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:42.558732 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80b01b7a-2c9f-41b5-ba92-9ade95a89cc8-openshift-service-ca" pod "80b01b7a-2c9f-41b5-ba92-9ade95a89cc8" (UID: "80b01b7a-2c9f-41b5-ba92-9ade95a89cc8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:42.566185 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:42.565908 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80b01b7a-2c9f-41b5-ba92-9ade95a89cc8-maas-api-tls" pod "80b01b7a-2c9f-41b5-ba92-9ade95a89cc8" (UID: "80b01b7a-2c9f-41b5-ba92-9ade95a89cc8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:42.597376 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:42.587558 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b01b7a-2c9f-41b5-ba92-9ade95a89cc8-kube-api-access-qrrw4" pod "80b01b7a-2c9f-41b5-ba92-9ade95a89cc8" (UID: "80b01b7a-2c9f-41b5-ba92-9ade95a89cc8"). InnerVolumeSpecName "kube-api-access-qrrw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:42.597376 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:42.592886 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80b01b7a-2c9f-41b5-ba92-9ade95a89cc8-metrics-certs" pod "80b01b7a-2c9f-41b5-ba92-9ade95a89cc8" (UID: "80b01b7a-2c9f-41b5-ba92-9ade95a89cc8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:42.658962 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:42.658921 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b01b7a-2c9f-41b5-ba92-9ade95a89cc8-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:42.659230 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:42.659209 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrrw4\" (UniqueName: \"kubernetes.io/projected/80b01b7a-2c9f-41b5-ba92-9ade95a89cc8-kube-api-access-qrrw4\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:42.659388 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:42.659371 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80b01b7a-2c9f-41b5-ba92-9ade95a89cc8-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:42.671073 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:42.671045 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80b01b7a-2c9f-41b5-ba92-9ade95a89cc8-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:43.265085 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:43.265059 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69" Oct 30 18:25:43.265606 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:43.265130 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69" event={"ID":"80b01b7a-2c9f-41b5-ba92-9ade95a89cc8","Type":"ContainerDied","Data":"2b5d039e5ead39731678cf31b3e0ef3c0bad75dbf31b3617655f27817a659d4f"} Oct 30 18:25:43.265748 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:43.265670 4455 scope.go:122] "RemoveContainer" containerID="c18a87dab12929ae8d261322054c87f35bc7585013af55fc369120b39423d8c4" Oct 30 18:25:43.307559 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:43.307525 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" event={"ID":"7894da9d-a98c-45ac-9968-bfe4e9747c49","Type":"ContainerStarted","Data":"ecccd275d82caf62c5c903c6bb4bfd1a8a1712ef2ca1485658f10801bba111a2"} Oct 30 18:25:43.307866 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:43.307833 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" event={"ID":"7894da9d-a98c-45ac-9968-bfe4e9747c49","Type":"ContainerStarted","Data":"2947968309e4d68198583c7aae8c870089087cef0bf82ac2cd3c7883844581db"} Oct 30 18:25:43.351688 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:43.351656 4455 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 18:25:43.351933 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:43.351901 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" podUID="cb055040-07c9-46eb-8649-dd1db076ee59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 18:25:43.366762 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:43.366701 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" podStartSLOduration=3.366682786 podStartE2EDuration="3.366682786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:43.363633187 +0000 UTC m=+1995.851270220" watchObservedRunningTime="2026-10-30 18:25:43.366682786 +0000 UTC m=+1995.854319822" Oct 30 18:25:43.430639 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:43.430611 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69"] Oct 30 18:25:43.435270 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:43.435245 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7ffac8-fb58688dc-zzr69"] Oct 30 18:25:43.447906 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:43.447876 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" Oct 30 18:25:43.448266 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:43.448243 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" Oct 30 18:25:43.449979 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:43.449949 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" Oct 30 18:25:43.646692 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:43.646617 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80b01b7a-2c9f-41b5-ba92-9ade95a89cc8" path="/var/lib/kubelet/pods/80b01b7a-2c9f-41b5-ba92-9ade95a89cc8/volumes" Oct 30 18:25:44.132760 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:44.132697 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" Oct 30 18:25:44.133994 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:44.133964 4455 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 18:25:44.134219 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:44.134192 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" podUID="7894da9d-a98c-45ac-9968-bfe4e9747c49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 18:25:44.351475 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:44.351445 4455 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 18:25:44.351711 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:44.351688 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" podUID="cb055040-07c9-46eb-8649-dd1db076ee59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 18:25:45.233395 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:45.220961 4455 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 18:25:45.233395 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:45.221021 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" podUID="7894da9d-a98c-45ac-9968-bfe4e9747c49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 18:25:45.353209 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:45.352773 4455 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 18:25:45.353209 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:45.352835 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" podUID="cb055040-07c9-46eb-8649-dd1db076ee59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 18:25:46.154652 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.153494 4455 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 18:25:46.154652 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.153556 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" podUID="7894da9d-a98c-45ac-9968-bfe4e9747c49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 18:25:46.354349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.351973 4455 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 18:25:46.354349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.352049 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" podUID="cb055040-07c9-46eb-8649-dd1db076ee59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 18:25:46.453821 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.453722 4455 generic.go:356] "Generic (PLEG): container finished" podID="ad790c3e-7a8f-462a-835c-870bb8f73002" containerID="44de37a7d527887cc6b6fb82249dd05aa09477668c5a67ffa5dbf65ce54152ab" exitCode=0 Oct 30 18:25:46.453821 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.453777 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" event={"ID":"ad790c3e-7a8f-462a-835c-870bb8f73002","Type":"ContainerDied","Data":"44de37a7d527887cc6b6fb82249dd05aa09477668c5a67ffa5dbf65ce54152ab"} Oct 30 18:25:46.554470 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.554445 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" Oct 30 18:25:46.664425 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.664389 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad790c3e-7a8f-462a-835c-870bb8f73002-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad790c3e-7a8f-462a-835c-870bb8f73002-istio-podinfo\") pod \"ad790c3e-7a8f-462a-835c-870bb8f73002\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " Oct 30 18:25:46.664601 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.664468 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad790c3e-7a8f-462a-835c-870bb8f73002-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad790c3e-7a8f-462a-835c-870bb8f73002-workload-socket\") pod \"ad790c3e-7a8f-462a-835c-870bb8f73002\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " Oct 30 18:25:46.664601 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.664527 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad790c3e-7a8f-462a-835c-870bb8f73002-istio-token\" (UniqueName: \"kubernetes.io/projected/ad790c3e-7a8f-462a-835c-870bb8f73002-istio-token\") pod \"ad790c3e-7a8f-462a-835c-870bb8f73002\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " Oct 30 18:25:46.664724 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.664702 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad790c3e-7a8f-462a-835c-870bb8f73002-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad790c3e-7a8f-462a-835c-870bb8f73002-workload-certs\") pod \"ad790c3e-7a8f-462a-835c-870bb8f73002\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " Oct 30 18:25:46.664788 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.664776 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad790c3e-7a8f-462a-835c-870bb8f73002-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad790c3e-7a8f-462a-835c-870bb8f73002-istio-data\") pod \"ad790c3e-7a8f-462a-835c-870bb8f73002\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " Oct 30 18:25:46.664864 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.664820 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad790c3e-7a8f-462a-835c-870bb8f73002-workload-socket" pod "ad790c3e-7a8f-462a-835c-870bb8f73002" (UID: "ad790c3e-7a8f-462a-835c-870bb8f73002"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:46.664935 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.664863 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad790c3e-7a8f-462a-835c-870bb8f73002-kube-api-access-zcw7q\" (UniqueName: \"kubernetes.io/projected/ad790c3e-7a8f-462a-835c-870bb8f73002-kube-api-access-zcw7q\") pod \"ad790c3e-7a8f-462a-835c-870bb8f73002\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " Oct 30 18:25:46.664996 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.664951 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad790c3e-7a8f-462a-835c-870bb8f73002-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad790c3e-7a8f-462a-835c-870bb8f73002-istio-envoy\") pod \"ad790c3e-7a8f-462a-835c-870bb8f73002\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " Oct 30 18:25:46.665089 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.665071 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad790c3e-7a8f-462a-835c-870bb8f73002-workload-certs" pod "ad790c3e-7a8f-462a-835c-870bb8f73002" (UID: "ad790c3e-7a8f-462a-835c-870bb8f73002"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:46.665163 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.665089 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad790c3e-7a8f-462a-835c-870bb8f73002-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad790c3e-7a8f-462a-835c-870bb8f73002-istiod-ca-cert\") pod \"ad790c3e-7a8f-462a-835c-870bb8f73002\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " Oct 30 18:25:46.665163 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.665141 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad790c3e-7a8f-462a-835c-870bb8f73002-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad790c3e-7a8f-462a-835c-870bb8f73002-credential-socket\") pod \"ad790c3e-7a8f-462a-835c-870bb8f73002\" (UID: \"ad790c3e-7a8f-462a-835c-870bb8f73002\") " Oct 30 18:25:46.665280 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.665089 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad790c3e-7a8f-462a-835c-870bb8f73002-istio-data" pod "ad790c3e-7a8f-462a-835c-870bb8f73002" (UID: "ad790c3e-7a8f-462a-835c-870bb8f73002"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:46.666171 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.665585 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad790c3e-7a8f-462a-835c-870bb8f73002-credential-socket" pod "ad790c3e-7a8f-462a-835c-870bb8f73002" (UID: "ad790c3e-7a8f-462a-835c-870bb8f73002"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:46.666171 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.665668 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad790c3e-7a8f-462a-835c-870bb8f73002-istiod-ca-cert" pod "ad790c3e-7a8f-462a-835c-870bb8f73002" (UID: "ad790c3e-7a8f-462a-835c-870bb8f73002"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:46.666171 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.666083 4455 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad790c3e-7a8f-462a-835c-870bb8f73002-istio-data\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:46.666171 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.666104 4455 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad790c3e-7a8f-462a-835c-870bb8f73002-istiod-ca-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:46.666171 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.666118 4455 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad790c3e-7a8f-462a-835c-870bb8f73002-credential-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:46.666171 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.666134 4455 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad790c3e-7a8f-462a-835c-870bb8f73002-workload-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:46.666171 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.666147 4455 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad790c3e-7a8f-462a-835c-870bb8f73002-workload-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:46.722957 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.696938 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad790c3e-7a8f-462a-835c-870bb8f73002-istio-envoy" pod "ad790c3e-7a8f-462a-835c-870bb8f73002" (UID: "ad790c3e-7a8f-462a-835c-870bb8f73002"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:46.744355 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.740497 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad790c3e-7a8f-462a-835c-870bb8f73002-kube-api-access-zcw7q" pod "ad790c3e-7a8f-462a-835c-870bb8f73002" (UID: "ad790c3e-7a8f-462a-835c-870bb8f73002"). InnerVolumeSpecName "kube-api-access-zcw7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:46.744355 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.742242 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad790c3e-7a8f-462a-835c-870bb8f73002-istio-token" pod "ad790c3e-7a8f-462a-835c-870bb8f73002" (UID: "ad790c3e-7a8f-462a-835c-870bb8f73002"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:46.745420 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.745389 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad790c3e-7a8f-462a-835c-870bb8f73002-istio-podinfo" pod "ad790c3e-7a8f-462a-835c-870bb8f73002" (UID: "ad790c3e-7a8f-462a-835c-870bb8f73002"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:25:46.802834 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.802727 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcw7q\" (UniqueName: \"kubernetes.io/projected/ad790c3e-7a8f-462a-835c-870bb8f73002-kube-api-access-zcw7q\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:46.802834 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.802771 4455 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad790c3e-7a8f-462a-835c-870bb8f73002-istio-envoy\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:46.802834 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.802787 4455 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad790c3e-7a8f-462a-835c-870bb8f73002-istio-podinfo\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:46.802834 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.802800 4455 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad790c3e-7a8f-462a-835c-870bb8f73002-istio-token\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:46.935650 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.935616 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-77db82-54bd486f48-6dv4g"] Oct 30 18:25:46.940131 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:46.940111 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-546655d4c5-knrgr"] Oct 30 18:25:47.145439 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:47.143481 4455 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 18:25:47.145439 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:47.143548 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" podUID="7894da9d-a98c-45ac-9968-bfe4e9747c49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 18:25:47.406400 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:47.396773 4455 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 18:25:47.406400 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:47.396835 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" podUID="cb055040-07c9-46eb-8649-dd1db076ee59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 18:25:48.015631 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:48.014814 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" event={"ID":"ad790c3e-7a8f-462a-835c-870bb8f73002","Type":"ContainerDied","Data":"a61df0114aa9aa1d00b6b3caef8e8ac10a0d0d600731e6a1a435e2eac9f316a2"} Oct 30 18:25:48.015631 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:48.014907 4455 scope.go:122] "RemoveContainer" containerID="44de37a7d527887cc6b6fb82249dd05aa09477668c5a67ffa5dbf65ce54152ab" Oct 30 18:25:48.015631 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:48.015126 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj" Oct 30 18:25:48.116346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:48.115937 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj"] Oct 30 18:25:48.131348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:48.125062 4455 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 18:25:48.131348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:48.125121 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" podUID="7894da9d-a98c-45ac-9968-bfe4e9747c49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 18:25:48.170348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:48.165661 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-8xnhj"] Oct 30 18:25:48.353798 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:48.353134 4455 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 18:25:48.353798 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:48.353273 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" podUID="cb055040-07c9-46eb-8649-dd1db076ee59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 18:25:49.122140 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:49.122103 4455 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 18:25:49.122692 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:49.122375 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" podUID="7894da9d-a98c-45ac-9968-bfe4e9747c49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 18:25:49.522810 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:49.518991 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" Oct 30 18:25:49.523430 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:49.523406 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" Oct 30 18:25:49.526761 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:49.526737 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" Oct 30 18:25:49.636489 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:49.636456 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad790c3e-7a8f-462a-835c-870bb8f73002" path="/var/lib/kubelet/pods/ad790c3e-7a8f-462a-835c-870bb8f73002/volumes" Oct 30 18:25:50.130056 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:50.130015 4455 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 18:25:50.131022 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:50.130991 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" podUID="7894da9d-a98c-45ac-9968-bfe4e9747c49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 18:25:50.485270 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:50.485233 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg"] Oct 30 18:25:50.486416 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:50.486363 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" podUID="a3c22942-b988-4208-ba9f-5b8becffa013" containerName="istio-proxy" containerID="cri-o://4c6dd46092dbffdeb7d9d59f59a2590d70169a1bab86aa8ad79a1ccb6020cbc7" gracePeriod=30 Oct 30 18:25:51.209958 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:51.209454 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" Oct 30 18:25:51.212624 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:51.212432 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" Oct 30 18:25:51.213788 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:51.213734 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" Oct 30 18:25:51.936474 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:51.936432 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-54bd486f48-6dv4g" podUID="30592a4b-d20f-44ab-bb1d-7bd5aaf9163d" containerName="payload-processing" containerID="cri-o://6d77307b5a4a24e88ac78ae2fed5f4c15842a29282e5c9032fed5fa7179ea4ff" gracePeriod=25 Oct 30 18:25:51.941012 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:51.940981 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-546655d4c5-knrgr" podUID="2f5acd91-9868-400b-b97f-f5416477f590" containerName="payload-pre-processing" containerID="cri-o://e06b4ba4ff79e5475bf94f36428dbef2a4109adb548f18d16a150ca91556bb13" gracePeriod=25 Oct 30 18:25:52.110693 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:52.110666 4455 generic.go:356] "Generic (PLEG): container finished" podID="30592a4b-d20f-44ab-bb1d-7bd5aaf9163d" containerID="6d77307b5a4a24e88ac78ae2fed5f4c15842a29282e5c9032fed5fa7179ea4ff" exitCode=0 Oct 30 18:25:52.110814 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:52.110743 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-54bd486f48-6dv4g" event={"ID":"30592a4b-d20f-44ab-bb1d-7bd5aaf9163d","Type":"ContainerDied","Data":"6d77307b5a4a24e88ac78ae2fed5f4c15842a29282e5c9032fed5fa7179ea4ff"} Oct 30 18:25:52.426884 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:52.419559 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-54bd486f48-6dv4g" Oct 30 18:25:52.483765 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:52.483731 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-546655d4c5-knrgr" Oct 30 18:25:52.592346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:52.585576 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f5acd91-9868-400b-b97f-f5416477f590-kube-api-access-qwtsd\" (UniqueName: \"kubernetes.io/projected/2f5acd91-9868-400b-b97f-f5416477f590-kube-api-access-qwtsd\") pod \"2f5acd91-9868-400b-b97f-f5416477f590\" (UID: \"2f5acd91-9868-400b-b97f-f5416477f590\") " Oct 30 18:25:52.592346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:52.585751 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30592a4b-d20f-44ab-bb1d-7bd5aaf9163d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30592a4b-d20f-44ab-bb1d-7bd5aaf9163d-plugins-config-volume\") pod \"30592a4b-d20f-44ab-bb1d-7bd5aaf9163d\" (UID: \"30592a4b-d20f-44ab-bb1d-7bd5aaf9163d\") " Oct 30 18:25:52.592346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:52.585894 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f5acd91-9868-400b-b97f-f5416477f590-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5acd91-9868-400b-b97f-f5416477f590-plugins-config-volume\") pod \"2f5acd91-9868-400b-b97f-f5416477f590\" (UID: \"2f5acd91-9868-400b-b97f-f5416477f590\") " Oct 30 18:25:52.592346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:52.586132 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30592a4b-d20f-44ab-bb1d-7bd5aaf9163d-kube-api-access-j6lkt\" (UniqueName: \"kubernetes.io/projected/30592a4b-d20f-44ab-bb1d-7bd5aaf9163d-kube-api-access-j6lkt\") pod \"30592a4b-d20f-44ab-bb1d-7bd5aaf9163d\" (UID: \"30592a4b-d20f-44ab-bb1d-7bd5aaf9163d\") " Oct 30 18:25:52.592346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:52.588264 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f5acd91-9868-400b-b97f-f5416477f590-plugins-config-volume" pod "2f5acd91-9868-400b-b97f-f5416477f590" (UID: "2f5acd91-9868-400b-b97f-f5416477f590"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:52.592346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:52.588911 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30592a4b-d20f-44ab-bb1d-7bd5aaf9163d-plugins-config-volume" pod "30592a4b-d20f-44ab-bb1d-7bd5aaf9163d" (UID: "30592a4b-d20f-44ab-bb1d-7bd5aaf9163d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:52.592346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:52.592185 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30592a4b-d20f-44ab-bb1d-7bd5aaf9163d-kube-api-access-j6lkt" pod "30592a4b-d20f-44ab-bb1d-7bd5aaf9163d" (UID: "30592a4b-d20f-44ab-bb1d-7bd5aaf9163d"). InnerVolumeSpecName "kube-api-access-j6lkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:52.631373 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:52.629725 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f5acd91-9868-400b-b97f-f5416477f590-kube-api-access-qwtsd" pod "2f5acd91-9868-400b-b97f-f5416477f590" (UID: "2f5acd91-9868-400b-b97f-f5416477f590"). InnerVolumeSpecName "kube-api-access-qwtsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:52.688739 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:52.688700 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6lkt\" (UniqueName: \"kubernetes.io/projected/30592a4b-d20f-44ab-bb1d-7bd5aaf9163d-kube-api-access-j6lkt\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:52.688739 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:52.688743 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwtsd\" (UniqueName: \"kubernetes.io/projected/2f5acd91-9868-400b-b97f-f5416477f590-kube-api-access-qwtsd\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:52.689001 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:52.688759 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30592a4b-d20f-44ab-bb1d-7bd5aaf9163d-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:52.689001 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:52.688773 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5acd91-9868-400b-b97f-f5416477f590-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:53.120402 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:53.120370 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-54bd486f48-6dv4g" Oct 30 18:25:53.120402 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:53.120382 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-77db82-54bd486f48-6dv4g" event={"ID":"30592a4b-d20f-44ab-bb1d-7bd5aaf9163d","Type":"ContainerDied","Data":"71840eef602bbd78ea0b016f7d9ce742fc894857af727c4d4656928b9e8f090e"} Oct 30 18:25:53.120634 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:53.120444 4455 scope.go:122] "RemoveContainer" containerID="6d77307b5a4a24e88ac78ae2fed5f4c15842a29282e5c9032fed5fa7179ea4ff" Oct 30 18:25:53.122024 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:53.121990 4455 generic.go:356] "Generic (PLEG): container finished" podID="2f5acd91-9868-400b-b97f-f5416477f590" containerID="e06b4ba4ff79e5475bf94f36428dbef2a4109adb548f18d16a150ca91556bb13" exitCode=0 Oct 30 18:25:53.122428 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:53.122401 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-546655d4c5-knrgr" event={"ID":"2f5acd91-9868-400b-b97f-f5416477f590","Type":"ContainerDied","Data":"e06b4ba4ff79e5475bf94f36428dbef2a4109adb548f18d16a150ca91556bb13"} Oct 30 18:25:53.122580 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:53.122559 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-546655d4c5-knrgr" event={"ID":"2f5acd91-9868-400b-b97f-f5416477f590","Type":"ContainerDied","Data":"0e81072bd7ae57ec1b69769b2feef53c3087e120c89001253c819398c9e3b01e"} Oct 30 18:25:53.122580 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:53.122444 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-546655d4c5-knrgr" Oct 30 18:25:53.136887 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:53.134204 4455 scope.go:122] "RemoveContainer" containerID="e06b4ba4ff79e5475bf94f36428dbef2a4109adb548f18d16a150ca91556bb13" Oct 30 18:25:53.150338 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:53.150292 4455 scope.go:122] "RemoveContainer" containerID="e06b4ba4ff79e5475bf94f36428dbef2a4109adb548f18d16a150ca91556bb13" Oct 30 18:25:53.150634 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:25:53.150614 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e06b4ba4ff79e5475bf94f36428dbef2a4109adb548f18d16a150ca91556bb13\": container with ID starting with e06b4ba4ff79e5475bf94f36428dbef2a4109adb548f18d16a150ca91556bb13 not found: ID does not exist" containerID="e06b4ba4ff79e5475bf94f36428dbef2a4109adb548f18d16a150ca91556bb13" Oct 30 18:25:53.150676 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:53.150643 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e06b4ba4ff79e5475bf94f36428dbef2a4109adb548f18d16a150ca91556bb13"} err="failed to get container status \"e06b4ba4ff79e5475bf94f36428dbef2a4109adb548f18d16a150ca91556bb13\": rpc error: code = NotFound desc = could not find container \"e06b4ba4ff79e5475bf94f36428dbef2a4109adb548f18d16a150ca91556bb13\": container with ID starting with e06b4ba4ff79e5475bf94f36428dbef2a4109adb548f18d16a150ca91556bb13 not found: ID does not exist" Oct 30 18:25:53.196658 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:53.196624 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-77db82-54bd486f48-6dv4g"] Oct 30 18:25:53.210877 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:53.210793 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-77db82-54bd486f48-6dv4g"] Oct 30 18:25:53.247643 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:53.247602 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-546655d4c5-knrgr"] Oct 30 18:25:53.251837 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:53.251808 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-77db82-546655d4c5-knrgr"] Oct 30 18:25:53.639857 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:53.639823 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f5acd91-9868-400b-b97f-f5416477f590" path="/var/lib/kubelet/pods/2f5acd91-9868-400b-b97f-f5416477f590/volumes" Oct 30 18:25:53.640544 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:53.640519 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30592a4b-d20f-44ab-bb1d-7bd5aaf9163d" path="/var/lib/kubelet/pods/30592a4b-d20f-44ab-bb1d-7bd5aaf9163d/volumes" Oct 30 18:25:55.832444 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:55.832375 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" Oct 30 18:25:55.844527 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:55.844497 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c22942-b988-4208-ba9f-5b8becffa013-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3c22942-b988-4208-ba9f-5b8becffa013-istio-data\") pod \"a3c22942-b988-4208-ba9f-5b8becffa013\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " Oct 30 18:25:55.844661 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:55.844546 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3c22942-b988-4208-ba9f-5b8becffa013-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3c22942-b988-4208-ba9f-5b8becffa013-istiod-ca-cert\") pod \"a3c22942-b988-4208-ba9f-5b8becffa013\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " Oct 30 18:25:55.844661 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:55.844598 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3c22942-b988-4208-ba9f-5b8becffa013-istio-token\" (UniqueName: \"kubernetes.io/projected/a3c22942-b988-4208-ba9f-5b8becffa013-istio-token\") pod \"a3c22942-b988-4208-ba9f-5b8becffa013\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " Oct 30 18:25:55.844661 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:55.844655 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c22942-b988-4208-ba9f-5b8becffa013-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3c22942-b988-4208-ba9f-5b8becffa013-istio-envoy\") pod \"a3c22942-b988-4208-ba9f-5b8becffa013\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " Oct 30 18:25:55.844838 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:55.844703 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c22942-b988-4208-ba9f-5b8becffa013-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c22942-b988-4208-ba9f-5b8becffa013-workload-socket\") pod \"a3c22942-b988-4208-ba9f-5b8becffa013\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " Oct 30 18:25:55.844838 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:55.844752 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c22942-b988-4208-ba9f-5b8becffa013-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3c22942-b988-4208-ba9f-5b8becffa013-workload-certs\") pod \"a3c22942-b988-4208-ba9f-5b8becffa013\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " Oct 30 18:25:55.844838 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:55.844798 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c22942-b988-4208-ba9f-5b8becffa013-istio-data" pod "a3c22942-b988-4208-ba9f-5b8becffa013" (UID: "a3c22942-b988-4208-ba9f-5b8becffa013"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:55.845003 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:55.844837 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3c22942-b988-4208-ba9f-5b8becffa013-kube-api-access-5xxrf\" (UniqueName: \"kubernetes.io/projected/a3c22942-b988-4208-ba9f-5b8becffa013-kube-api-access-5xxrf\") pod \"a3c22942-b988-4208-ba9f-5b8becffa013\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " Oct 30 18:25:55.845003 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:55.844866 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3c22942-b988-4208-ba9f-5b8becffa013-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3c22942-b988-4208-ba9f-5b8becffa013-istio-podinfo\") pod \"a3c22942-b988-4208-ba9f-5b8becffa013\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " Oct 30 18:25:55.845003 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:55.844970 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c22942-b988-4208-ba9f-5b8becffa013-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c22942-b988-4208-ba9f-5b8becffa013-credential-socket\") pod \"a3c22942-b988-4208-ba9f-5b8becffa013\" (UID: \"a3c22942-b988-4208-ba9f-5b8becffa013\") " Oct 30 18:25:55.845154 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:55.845011 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c22942-b988-4208-ba9f-5b8becffa013-workload-socket" pod "a3c22942-b988-4208-ba9f-5b8becffa013" (UID: "a3c22942-b988-4208-ba9f-5b8becffa013"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:55.845154 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:55.845060 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c22942-b988-4208-ba9f-5b8becffa013-workload-certs" pod "a3c22942-b988-4208-ba9f-5b8becffa013" (UID: "a3c22942-b988-4208-ba9f-5b8becffa013"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:55.845154 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:55.845080 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3c22942-b988-4208-ba9f-5b8becffa013-istiod-ca-cert" pod "a3c22942-b988-4208-ba9f-5b8becffa013" (UID: "a3c22942-b988-4208-ba9f-5b8becffa013"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:55.845368 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:55.845314 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c22942-b988-4208-ba9f-5b8becffa013-credential-socket" pod "a3c22942-b988-4208-ba9f-5b8becffa013" (UID: "a3c22942-b988-4208-ba9f-5b8becffa013"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:55.845689 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:55.845667 4455 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3c22942-b988-4208-ba9f-5b8becffa013-istio-data\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:55.845689 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:55.845689 4455 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3c22942-b988-4208-ba9f-5b8becffa013-istiod-ca-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:55.845880 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:55.845697 4455 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c22942-b988-4208-ba9f-5b8becffa013-workload-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:55.845880 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:55.845705 4455 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3c22942-b988-4208-ba9f-5b8becffa013-workload-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:55.845880 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:55.845714 4455 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c22942-b988-4208-ba9f-5b8becffa013-credential-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:55.847013 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:55.846991 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c22942-b988-4208-ba9f-5b8becffa013-istio-envoy" pod "a3c22942-b988-4208-ba9f-5b8becffa013" (UID: "a3c22942-b988-4208-ba9f-5b8becffa013"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:55.847207 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:55.847186 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3c22942-b988-4208-ba9f-5b8becffa013-istio-token" pod "a3c22942-b988-4208-ba9f-5b8becffa013" (UID: "a3c22942-b988-4208-ba9f-5b8becffa013"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:55.847632 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:55.847613 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3c22942-b988-4208-ba9f-5b8becffa013-istio-podinfo" pod "a3c22942-b988-4208-ba9f-5b8becffa013" (UID: "a3c22942-b988-4208-ba9f-5b8becffa013"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:25:55.847762 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:55.847742 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3c22942-b988-4208-ba9f-5b8becffa013-kube-api-access-5xxrf" pod "a3c22942-b988-4208-ba9f-5b8becffa013" (UID: "a3c22942-b988-4208-ba9f-5b8becffa013"). InnerVolumeSpecName "kube-api-access-5xxrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:55.947449 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:55.947414 4455 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3c22942-b988-4208-ba9f-5b8becffa013-istio-token\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:55.947449 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:55.947441 4455 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3c22942-b988-4208-ba9f-5b8becffa013-istio-envoy\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:55.947449 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:55.947451 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xxrf\" (UniqueName: \"kubernetes.io/projected/a3c22942-b988-4208-ba9f-5b8becffa013-kube-api-access-5xxrf\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:55.947449 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:55.947460 4455 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3c22942-b988-4208-ba9f-5b8becffa013-istio-podinfo\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:25:56.149432 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:56.149333 4455 generic.go:356] "Generic (PLEG): container finished" podID="a3c22942-b988-4208-ba9f-5b8becffa013" containerID="4c6dd46092dbffdeb7d9d59f59a2590d70169a1bab86aa8ad79a1ccb6020cbc7" exitCode=0 Oct 30 18:25:56.149432 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:56.149389 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" event={"ID":"a3c22942-b988-4208-ba9f-5b8becffa013","Type":"ContainerDied","Data":"4c6dd46092dbffdeb7d9d59f59a2590d70169a1bab86aa8ad79a1ccb6020cbc7"} Oct 30 18:25:56.149688 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:56.149430 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" Oct 30 18:25:56.149688 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:56.149450 4455 scope.go:122] "RemoveContainer" containerID="4c6dd46092dbffdeb7d9d59f59a2590d70169a1bab86aa8ad79a1ccb6020cbc7" Oct 30 18:25:56.149688 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:56.149437 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg" event={"ID":"a3c22942-b988-4208-ba9f-5b8becffa013","Type":"ContainerDied","Data":"8b1ac3c2403681c325b12f7620f895af847066925be36864a9a8ceca6e92e2bb"} Oct 30 18:25:56.165214 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:56.165187 4455 scope.go:122] "RemoveContainer" containerID="4c6dd46092dbffdeb7d9d59f59a2590d70169a1bab86aa8ad79a1ccb6020cbc7" Oct 30 18:25:56.165543 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:25:56.165515 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c6dd46092dbffdeb7d9d59f59a2590d70169a1bab86aa8ad79a1ccb6020cbc7\": container with ID starting with 4c6dd46092dbffdeb7d9d59f59a2590d70169a1bab86aa8ad79a1ccb6020cbc7 not found: ID does not exist" containerID="4c6dd46092dbffdeb7d9d59f59a2590d70169a1bab86aa8ad79a1ccb6020cbc7" Oct 30 18:25:56.165641 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:56.165553 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c6dd46092dbffdeb7d9d59f59a2590d70169a1bab86aa8ad79a1ccb6020cbc7"} err="failed to get container status \"4c6dd46092dbffdeb7d9d59f59a2590d70169a1bab86aa8ad79a1ccb6020cbc7\": rpc error: code = NotFound desc = could not find container \"4c6dd46092dbffdeb7d9d59f59a2590d70169a1bab86aa8ad79a1ccb6020cbc7\": container with ID starting with 4c6dd46092dbffdeb7d9d59f59a2590d70169a1bab86aa8ad79a1ccb6020cbc7 not found: ID does not exist" Oct 30 18:25:56.213148 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:56.213111 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg"] Oct 30 18:25:56.221768 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:56.221734 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-zndwg"] Oct 30 18:25:57.651355 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:57.651146 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3c22942-b988-4208-ba9f-5b8becffa013" path="/var/lib/kubelet/pods/a3c22942-b988-4208-ba9f-5b8becffa013/volumes" Oct 30 18:25:59.232686 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:59.232656 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ca5f33-54dcbfd498-fvclr"] Oct 30 18:25:59.268800 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:25:59.268767 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ca5f33-546df68d7-nnnlz"] Oct 30 18:26:00.402347 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:26:00.401963 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:26:04.233752 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:04.233697 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca5f33-54dcbfd498-fvclr" podUID="8a49c1e9-b602-4d17-828c-a95ea3325c66" containerName="payload-processing" containerID="cri-o://a8634a97c74738f82182a3706e5f6552b14d1f7d1ecb1b4046458f83c3a1d722" gracePeriod=25 Oct 30 18:26:04.269235 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:04.269197 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca5f33-546df68d7-nnnlz" podUID="0fdeba52-780c-4631-aded-74a165c8c700" containerName="payload-pre-processing" containerID="cri-o://59583ad94991e7462c807ce161969310c06f45dd69992cdb6dd28e7697f843f2" gracePeriod=25 Oct 30 18:26:04.578703 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:04.578673 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca5f33-54dcbfd498-fvclr" Oct 30 18:26:04.618613 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:04.618588 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca5f33-546df68d7-nnnlz" Oct 30 18:26:04.654622 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:04.654586 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a49c1e9-b602-4d17-828c-a95ea3325c66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a49c1e9-b602-4d17-828c-a95ea3325c66-plugins-config-volume\") pod \"8a49c1e9-b602-4d17-828c-a95ea3325c66\" (UID: \"8a49c1e9-b602-4d17-828c-a95ea3325c66\") " Oct 30 18:26:04.654866 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:04.654849 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a49c1e9-b602-4d17-828c-a95ea3325c66-kube-api-access-bzct7\" (UniqueName: \"kubernetes.io/projected/8a49c1e9-b602-4d17-828c-a95ea3325c66-kube-api-access-bzct7\") pod \"8a49c1e9-b602-4d17-828c-a95ea3325c66\" (UID: \"8a49c1e9-b602-4d17-828c-a95ea3325c66\") " Oct 30 18:26:04.654976 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:04.654951 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a49c1e9-b602-4d17-828c-a95ea3325c66-plugins-config-volume" pod "8a49c1e9-b602-4d17-828c-a95ea3325c66" (UID: "8a49c1e9-b602-4d17-828c-a95ea3325c66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:04.655495 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:04.655436 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a49c1e9-b602-4d17-828c-a95ea3325c66-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:26:04.657075 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:04.657054 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a49c1e9-b602-4d17-828c-a95ea3325c66-kube-api-access-bzct7" pod "8a49c1e9-b602-4d17-828c-a95ea3325c66" (UID: "8a49c1e9-b602-4d17-828c-a95ea3325c66"). InnerVolumeSpecName "kube-api-access-bzct7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:04.770896 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:04.770855 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fdeba52-780c-4631-aded-74a165c8c700-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fdeba52-780c-4631-aded-74a165c8c700-plugins-config-volume\") pod \"0fdeba52-780c-4631-aded-74a165c8c700\" (UID: \"0fdeba52-780c-4631-aded-74a165c8c700\") " Oct 30 18:26:04.787503 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:04.771795 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fdeba52-780c-4631-aded-74a165c8c700-plugins-config-volume" pod "0fdeba52-780c-4631-aded-74a165c8c700" (UID: "0fdeba52-780c-4631-aded-74a165c8c700"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:04.787854 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:04.787823 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fdeba52-780c-4631-aded-74a165c8c700-kube-api-access-qp6nc\" (UniqueName: \"kubernetes.io/projected/0fdeba52-780c-4631-aded-74a165c8c700-kube-api-access-qp6nc\") pod \"0fdeba52-780c-4631-aded-74a165c8c700\" (UID: \"0fdeba52-780c-4631-aded-74a165c8c700\") " Oct 30 18:26:04.794413 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:04.788723 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzct7\" (UniqueName: \"kubernetes.io/projected/8a49c1e9-b602-4d17-828c-a95ea3325c66-kube-api-access-bzct7\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:26:04.794413 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:04.788751 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fdeba52-780c-4631-aded-74a165c8c700-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:26:04.802372 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:04.799730 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fdeba52-780c-4631-aded-74a165c8c700-kube-api-access-qp6nc" pod "0fdeba52-780c-4631-aded-74a165c8c700" (UID: "0fdeba52-780c-4631-aded-74a165c8c700"). InnerVolumeSpecName "kube-api-access-qp6nc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:04.890154 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:04.890120 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp6nc\" (UniqueName: \"kubernetes.io/projected/0fdeba52-780c-4631-aded-74a165c8c700-kube-api-access-qp6nc\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:26:05.285464 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:05.285425 4455 generic.go:356] "Generic (PLEG): container finished" podID="8a49c1e9-b602-4d17-828c-a95ea3325c66" containerID="a8634a97c74738f82182a3706e5f6552b14d1f7d1ecb1b4046458f83c3a1d722" exitCode=0 Oct 30 18:26:05.285912 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:05.285474 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca5f33-54dcbfd498-fvclr" Oct 30 18:26:05.285912 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:05.285489 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca5f33-54dcbfd498-fvclr" event={"ID":"8a49c1e9-b602-4d17-828c-a95ea3325c66","Type":"ContainerDied","Data":"a8634a97c74738f82182a3706e5f6552b14d1f7d1ecb1b4046458f83c3a1d722"} Oct 30 18:26:05.285912 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:05.285550 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca5f33-54dcbfd498-fvclr" event={"ID":"8a49c1e9-b602-4d17-828c-a95ea3325c66","Type":"ContainerDied","Data":"376138a0a8d46553898610979e1fd95ed07bdae7feeb6f7e9aa5942ae985327c"} Oct 30 18:26:05.285912 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:05.285579 4455 scope.go:122] "RemoveContainer" containerID="a8634a97c74738f82182a3706e5f6552b14d1f7d1ecb1b4046458f83c3a1d722" Oct 30 18:26:05.287555 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:05.287528 4455 generic.go:356] "Generic (PLEG): container finished" podID="0fdeba52-780c-4631-aded-74a165c8c700" containerID="59583ad94991e7462c807ce161969310c06f45dd69992cdb6dd28e7697f843f2" exitCode=0 Oct 30 18:26:05.287672 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:05.287576 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca5f33-546df68d7-nnnlz" event={"ID":"0fdeba52-780c-4631-aded-74a165c8c700","Type":"ContainerDied","Data":"59583ad94991e7462c807ce161969310c06f45dd69992cdb6dd28e7697f843f2"} Oct 30 18:26:05.287672 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:05.287594 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca5f33-546df68d7-nnnlz" Oct 30 18:26:05.287672 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:05.287628 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca5f33-546df68d7-nnnlz" event={"ID":"0fdeba52-780c-4631-aded-74a165c8c700","Type":"ContainerDied","Data":"87cc4ca087d4a94305fede38ac4615738533055b34e2b67169676ef9628e622a"} Oct 30 18:26:05.299024 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:05.299006 4455 scope.go:122] "RemoveContainer" containerID="a8634a97c74738f82182a3706e5f6552b14d1f7d1ecb1b4046458f83c3a1d722" Oct 30 18:26:05.299311 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:26:05.299288 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8634a97c74738f82182a3706e5f6552b14d1f7d1ecb1b4046458f83c3a1d722\": container with ID starting with a8634a97c74738f82182a3706e5f6552b14d1f7d1ecb1b4046458f83c3a1d722 not found: ID does not exist" containerID="a8634a97c74738f82182a3706e5f6552b14d1f7d1ecb1b4046458f83c3a1d722" Oct 30 18:26:05.299408 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:05.299338 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8634a97c74738f82182a3706e5f6552b14d1f7d1ecb1b4046458f83c3a1d722"} err="failed to get container status \"a8634a97c74738f82182a3706e5f6552b14d1f7d1ecb1b4046458f83c3a1d722\": rpc error: code = NotFound desc = could not find container \"a8634a97c74738f82182a3706e5f6552b14d1f7d1ecb1b4046458f83c3a1d722\": container with ID starting with a8634a97c74738f82182a3706e5f6552b14d1f7d1ecb1b4046458f83c3a1d722 not found: ID does not exist" Oct 30 18:26:05.299408 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:05.299361 4455 scope.go:122] "RemoveContainer" containerID="59583ad94991e7462c807ce161969310c06f45dd69992cdb6dd28e7697f843f2" Oct 30 18:26:05.311579 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:05.311552 4455 scope.go:122] "RemoveContainer" containerID="59583ad94991e7462c807ce161969310c06f45dd69992cdb6dd28e7697f843f2" Oct 30 18:26:05.311875 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:26:05.311857 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59583ad94991e7462c807ce161969310c06f45dd69992cdb6dd28e7697f843f2\": container with ID starting with 59583ad94991e7462c807ce161969310c06f45dd69992cdb6dd28e7697f843f2 not found: ID does not exist" containerID="59583ad94991e7462c807ce161969310c06f45dd69992cdb6dd28e7697f843f2" Oct 30 18:26:05.311926 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:05.311883 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59583ad94991e7462c807ce161969310c06f45dd69992cdb6dd28e7697f843f2"} err="failed to get container status \"59583ad94991e7462c807ce161969310c06f45dd69992cdb6dd28e7697f843f2\": rpc error: code = NotFound desc = could not find container \"59583ad94991e7462c807ce161969310c06f45dd69992cdb6dd28e7697f843f2\": container with ID starting with 59583ad94991e7462c807ce161969310c06f45dd69992cdb6dd28e7697f843f2 not found: ID does not exist" Oct 30 18:26:05.403403 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:05.403368 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ca5f33-54dcbfd498-fvclr"] Oct 30 18:26:05.448073 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:05.448036 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ca5f33-54dcbfd498-fvclr"] Oct 30 18:26:05.478464 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:05.478413 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ca5f33-546df68d7-nnnlz"] Oct 30 18:26:05.500487 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:05.500447 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ca5f33-546df68d7-nnnlz"] Oct 30 18:26:05.637271 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:05.637188 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fdeba52-780c-4631-aded-74a165c8c700" path="/var/lib/kubelet/pods/0fdeba52-780c-4631-aded-74a165c8c700/volumes" Oct 30 18:26:05.637816 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:05.637794 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a49c1e9-b602-4d17-828c-a95ea3325c66" path="/var/lib/kubelet/pods/8a49c1e9-b602-4d17-828c-a95ea3325c66/volumes" Oct 30 18:26:10.132721 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:10.132643 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1fae38-d874c668-m42lh"] Oct 30 18:26:10.169288 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:10.169258 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1fae38-6b77fbf575-d99xj"] Oct 30 18:26:12.173650 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:12.173602 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q"] Oct 30 18:26:12.174195 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:12.173880 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q" podUID="13d35525-5ffe-467e-a701-3200ca4facfd" containerName="maas-api" containerID="cri-o://bd27d377b75f2037c54c0198f100967d5da6a89954de07f9481871bf800926cf" gracePeriod=30 Oct 30 18:26:12.352703 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:12.352659 4455 generic.go:356] "Generic (PLEG): container finished" podID="13d35525-5ffe-467e-a701-3200ca4facfd" containerID="bd27d377b75f2037c54c0198f100967d5da6a89954de07f9481871bf800926cf" exitCode=0 Oct 30 18:26:12.352889 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:12.352759 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q" event={"ID":"13d35525-5ffe-467e-a701-3200ca4facfd","Type":"ContainerDied","Data":"bd27d377b75f2037c54c0198f100967d5da6a89954de07f9481871bf800926cf"} Oct 30 18:26:12.908609 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:12.908061 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q" Oct 30 18:26:12.998510 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:12.998478 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13d35525-5ffe-467e-a701-3200ca4facfd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d35525-5ffe-467e-a701-3200ca4facfd-maas-api-tls\") pod \"13d35525-5ffe-467e-a701-3200ca4facfd\" (UID: \"13d35525-5ffe-467e-a701-3200ca4facfd\") " Oct 30 18:26:12.998703 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:12.998518 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13d35525-5ffe-467e-a701-3200ca4facfd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d35525-5ffe-467e-a701-3200ca4facfd-metrics-certs\") pod \"13d35525-5ffe-467e-a701-3200ca4facfd\" (UID: \"13d35525-5ffe-467e-a701-3200ca4facfd\") " Oct 30 18:26:12.998703 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:12.998620 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13d35525-5ffe-467e-a701-3200ca4facfd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d35525-5ffe-467e-a701-3200ca4facfd-openshift-service-ca\") pod \"13d35525-5ffe-467e-a701-3200ca4facfd\" (UID: \"13d35525-5ffe-467e-a701-3200ca4facfd\") " Oct 30 18:26:12.998830 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:12.998811 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13d35525-5ffe-467e-a701-3200ca4facfd-kube-api-access-m8ct7\" (UniqueName: \"kubernetes.io/projected/13d35525-5ffe-467e-a701-3200ca4facfd-kube-api-access-m8ct7\") pod \"13d35525-5ffe-467e-a701-3200ca4facfd\" (UID: \"13d35525-5ffe-467e-a701-3200ca4facfd\") " Oct 30 18:26:12.999027 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:12.998994 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13d35525-5ffe-467e-a701-3200ca4facfd-openshift-service-ca" pod "13d35525-5ffe-467e-a701-3200ca4facfd" (UID: "13d35525-5ffe-467e-a701-3200ca4facfd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:12.999738 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:12.999712 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d35525-5ffe-467e-a701-3200ca4facfd-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:26:13.000999 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:13.000976 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13d35525-5ffe-467e-a701-3200ca4facfd-maas-api-tls" pod "13d35525-5ffe-467e-a701-3200ca4facfd" (UID: "13d35525-5ffe-467e-a701-3200ca4facfd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:13.001101 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:13.001029 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13d35525-5ffe-467e-a701-3200ca4facfd-metrics-certs" pod "13d35525-5ffe-467e-a701-3200ca4facfd" (UID: "13d35525-5ffe-467e-a701-3200ca4facfd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:13.001194 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:13.001175 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d35525-5ffe-467e-a701-3200ca4facfd-kube-api-access-m8ct7" pod "13d35525-5ffe-467e-a701-3200ca4facfd" (UID: "13d35525-5ffe-467e-a701-3200ca4facfd"). InnerVolumeSpecName "kube-api-access-m8ct7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:13.128783 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:13.128746 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8ct7\" (UniqueName: \"kubernetes.io/projected/13d35525-5ffe-467e-a701-3200ca4facfd-kube-api-access-m8ct7\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:26:13.129019 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:13.129003 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d35525-5ffe-467e-a701-3200ca4facfd-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:26:13.129102 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:13.129093 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d35525-5ffe-467e-a701-3200ca4facfd-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:26:13.361979 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:13.361947 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q" Oct 30 18:26:13.361979 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:13.361962 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q" event={"ID":"13d35525-5ffe-467e-a701-3200ca4facfd","Type":"ContainerDied","Data":"1eae9bd612d315f0bd50925b8ea58e3be0da8656ed07f9fce0f81d8ffc2ba4cd"} Oct 30 18:26:13.362491 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:13.362021 4455 scope.go:122] "RemoveContainer" containerID="bd27d377b75f2037c54c0198f100967d5da6a89954de07f9481871bf800926cf" Oct 30 18:26:13.420110 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:13.420072 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q"] Oct 30 18:26:13.433275 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:13.433243 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q"] Oct 30 18:26:13.635532 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:13.635453 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13d35525-5ffe-467e-a701-3200ca4facfd" path="/var/lib/kubelet/pods/13d35525-5ffe-467e-a701-3200ca4facfd/volumes" Oct 30 18:26:15.133875 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:15.133834 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-1fae38-d874c668-m42lh" podUID="c436ce58-c966-4069-bbe3-747b16908679" containerName="payload-processing" containerID="cri-o://20b958598c2497793f2163aaf8afa2c2cbe56a1cca952b928f3a5468ebb81312" gracePeriod=25 Oct 30 18:26:15.170123 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:15.170090 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1fae38-6b77fbf575-d99xj" podUID="d14b8e74-04c8-4948-8e26-12c06bc39341" containerName="payload-pre-processing" containerID="cri-o://8261d8c77fb921026e092e09fc902eaa0bbdb74b86ff7371d0bd2ba03005e6dc" gracePeriod=25 Oct 30 18:26:15.451233 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:15.451190 4455 generic.go:356] "Generic (PLEG): container finished" podID="d14b8e74-04c8-4948-8e26-12c06bc39341" containerID="8261d8c77fb921026e092e09fc902eaa0bbdb74b86ff7371d0bd2ba03005e6dc" exitCode=0 Oct 30 18:26:15.451425 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:15.451389 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1fae38-6b77fbf575-d99xj" event={"ID":"d14b8e74-04c8-4948-8e26-12c06bc39341","Type":"ContainerDied","Data":"8261d8c77fb921026e092e09fc902eaa0bbdb74b86ff7371d0bd2ba03005e6dc"} Oct 30 18:26:15.463198 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:15.457990 4455 generic.go:356] "Generic (PLEG): container finished" podID="c436ce58-c966-4069-bbe3-747b16908679" containerID="20b958598c2497793f2163aaf8afa2c2cbe56a1cca952b928f3a5468ebb81312" exitCode=0 Oct 30 18:26:15.463198 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:15.458080 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1fae38-d874c668-m42lh" event={"ID":"c436ce58-c966-4069-bbe3-747b16908679","Type":"ContainerDied","Data":"20b958598c2497793f2163aaf8afa2c2cbe56a1cca952b928f3a5468ebb81312"} Oct 30 18:26:15.485038 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:15.485009 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1fae38-d874c668-m42lh" Oct 30 18:26:15.511591 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:15.511566 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1fae38-6b77fbf575-d99xj" Oct 30 18:26:15.581616 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:15.581092 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c436ce58-c966-4069-bbe3-747b16908679-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c436ce58-c966-4069-bbe3-747b16908679-plugins-config-volume\") pod \"c436ce58-c966-4069-bbe3-747b16908679\" (UID: \"c436ce58-c966-4069-bbe3-747b16908679\") " Oct 30 18:26:15.581616 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:15.581216 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c436ce58-c966-4069-bbe3-747b16908679-kube-api-access-flbng\" (UniqueName: \"kubernetes.io/projected/c436ce58-c966-4069-bbe3-747b16908679-kube-api-access-flbng\") pod \"c436ce58-c966-4069-bbe3-747b16908679\" (UID: \"c436ce58-c966-4069-bbe3-747b16908679\") " Oct 30 18:26:15.581616 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:15.581273 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d14b8e74-04c8-4948-8e26-12c06bc39341-kube-api-access-ckp97\" (UniqueName: \"kubernetes.io/projected/d14b8e74-04c8-4948-8e26-12c06bc39341-kube-api-access-ckp97\") pod \"d14b8e74-04c8-4948-8e26-12c06bc39341\" (UID: \"d14b8e74-04c8-4948-8e26-12c06bc39341\") " Oct 30 18:26:15.581616 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:15.581428 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d14b8e74-04c8-4948-8e26-12c06bc39341-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d14b8e74-04c8-4948-8e26-12c06bc39341-plugins-config-volume\") pod \"d14b8e74-04c8-4948-8e26-12c06bc39341\" (UID: \"d14b8e74-04c8-4948-8e26-12c06bc39341\") " Oct 30 18:26:15.582957 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:15.582488 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d14b8e74-04c8-4948-8e26-12c06bc39341-plugins-config-volume" pod "d14b8e74-04c8-4948-8e26-12c06bc39341" (UID: "d14b8e74-04c8-4948-8e26-12c06bc39341"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:15.582957 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:15.582880 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c436ce58-c966-4069-bbe3-747b16908679-plugins-config-volume" pod "c436ce58-c966-4069-bbe3-747b16908679" (UID: "c436ce58-c966-4069-bbe3-747b16908679"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:15.591754 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:15.591501 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c436ce58-c966-4069-bbe3-747b16908679-kube-api-access-flbng" pod "c436ce58-c966-4069-bbe3-747b16908679" (UID: "c436ce58-c966-4069-bbe3-747b16908679"). InnerVolumeSpecName "kube-api-access-flbng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:15.597394 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:15.595890 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d14b8e74-04c8-4948-8e26-12c06bc39341-kube-api-access-ckp97" pod "d14b8e74-04c8-4948-8e26-12c06bc39341" (UID: "d14b8e74-04c8-4948-8e26-12c06bc39341"). InnerVolumeSpecName "kube-api-access-ckp97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:15.648578 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:15.648514 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1fae38-5f87cf55cc-98m2q" podUID="13d35525-5ffe-467e-a701-3200ca4facfd" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.169:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:26:15.683896 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:15.683811 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c436ce58-c966-4069-bbe3-747b16908679-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:26:15.683896 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:15.683841 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flbng\" (UniqueName: \"kubernetes.io/projected/c436ce58-c966-4069-bbe3-747b16908679-kube-api-access-flbng\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:26:15.683896 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:15.683854 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckp97\" (UniqueName: \"kubernetes.io/projected/d14b8e74-04c8-4948-8e26-12c06bc39341-kube-api-access-ckp97\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:26:15.683896 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:15.683865 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d14b8e74-04c8-4948-8e26-12c06bc39341-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:26:16.507548 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:16.507510 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1fae38-6b77fbf575-d99xj" event={"ID":"d14b8e74-04c8-4948-8e26-12c06bc39341","Type":"ContainerDied","Data":"79ca4e3cb145ba86ecd998872533635904d71681b4707141e30b9f4fd12c60b9"} Oct 30 18:26:16.508149 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:16.508130 4455 scope.go:122] "RemoveContainer" containerID="8261d8c77fb921026e092e09fc902eaa0bbdb74b86ff7371d0bd2ba03005e6dc" Oct 30 18:26:16.508441 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:16.508427 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1fae38-6b77fbf575-d99xj" Oct 30 18:26:16.519781 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:16.518740 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1fae38-d874c668-m42lh" event={"ID":"c436ce58-c966-4069-bbe3-747b16908679","Type":"ContainerDied","Data":"75bab5d1874cb0b0a02a54e2e595961ac303ee276dd9e9a35885c3851351af40"} Oct 30 18:26:16.519781 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:16.518858 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1fae38-d874c668-m42lh" Oct 30 18:26:16.562415 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:16.553819 4455 scope.go:122] "RemoveContainer" containerID="20b958598c2497793f2163aaf8afa2c2cbe56a1cca952b928f3a5468ebb81312" Oct 30 18:26:16.730266 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:16.725906 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1fae38-d874c668-m42lh"] Oct 30 18:26:16.817135 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:16.816421 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1fae38-d874c668-m42lh"] Oct 30 18:26:16.817135 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:16.816497 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1fae38-6b77fbf575-d99xj"] Oct 30 18:26:16.821973 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:16.821939 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1fae38-6b77fbf575-d99xj"] Oct 30 18:26:17.689963 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:17.689933 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c436ce58-c966-4069-bbe3-747b16908679" path="/var/lib/kubelet/pods/c436ce58-c966-4069-bbe3-747b16908679/volumes" Oct 30 18:26:17.691100 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:17.691011 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d14b8e74-04c8-4948-8e26-12c06bc39341" path="/var/lib/kubelet/pods/d14b8e74-04c8-4948-8e26-12c06bc39341/volumes" Oct 30 18:26:25.837667 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.837627 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9"] Oct 30 18:26:25.839375 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.839348 4455 cpu_manager.go:414] "containerMap: removing container" podUID="13d35525-5ffe-467e-a701-3200ca4facfd" containerName="maas-api" Oct 30 18:26:25.839375 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.839374 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13d35525-5ffe-467e-a701-3200ca4facfd" containerName="maas-api" Oct 30 18:26:25.839538 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.839425 4455 cpu_manager.go:414] "containerMap: removing container" podUID="d14b8e74-04c8-4948-8e26-12c06bc39341" containerName="payload-pre-processing" Oct 30 18:26:25.839538 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.839434 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d14b8e74-04c8-4948-8e26-12c06bc39341" containerName="payload-pre-processing" Oct 30 18:26:25.839538 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.839451 4455 cpu_manager.go:414] "containerMap: removing container" podUID="c436ce58-c966-4069-bbe3-747b16908679" containerName="payload-processing" Oct 30 18:26:25.839538 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.839459 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c436ce58-c966-4069-bbe3-747b16908679" containerName="payload-processing" Oct 30 18:26:25.839538 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.839503 4455 cpu_manager.go:414] "containerMap: removing container" podUID="80b01b7a-2c9f-41b5-ba92-9ade95a89cc8" containerName="maas-api" Oct 30 18:26:25.839538 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.839510 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80b01b7a-2c9f-41b5-ba92-9ade95a89cc8" containerName="maas-api" Oct 30 18:26:25.839538 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.839525 4455 cpu_manager.go:414] "containerMap: removing container" podUID="ad790c3e-7a8f-462a-835c-870bb8f73002" containerName="istio-proxy" Oct 30 18:26:25.839538 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.839531 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad790c3e-7a8f-462a-835c-870bb8f73002" containerName="istio-proxy" Oct 30 18:26:25.839785 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.839548 4455 cpu_manager.go:414] "containerMap: removing container" podUID="0fdeba52-780c-4631-aded-74a165c8c700" containerName="payload-pre-processing" Oct 30 18:26:25.839785 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.839556 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fdeba52-780c-4631-aded-74a165c8c700" containerName="payload-pre-processing" Oct 30 18:26:25.839785 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.839569 4455 cpu_manager.go:414] "containerMap: removing container" podUID="8a49c1e9-b602-4d17-828c-a95ea3325c66" containerName="payload-processing" Oct 30 18:26:25.839785 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.839576 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a49c1e9-b602-4d17-828c-a95ea3325c66" containerName="payload-processing" Oct 30 18:26:25.839785 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.839596 4455 cpu_manager.go:414] "containerMap: removing container" podUID="80b01b7a-2c9f-41b5-ba92-9ade95a89cc8" containerName="maas-api" Oct 30 18:26:25.839785 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.839602 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80b01b7a-2c9f-41b5-ba92-9ade95a89cc8" containerName="maas-api" Oct 30 18:26:25.839785 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.839617 4455 cpu_manager.go:414] "containerMap: removing container" podUID="30592a4b-d20f-44ab-bb1d-7bd5aaf9163d" containerName="payload-processing" Oct 30 18:26:25.839785 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.839623 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30592a4b-d20f-44ab-bb1d-7bd5aaf9163d" containerName="payload-processing" Oct 30 18:26:25.839785 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.839642 4455 cpu_manager.go:414] "containerMap: removing container" podUID="2f5acd91-9868-400b-b97f-f5416477f590" containerName="payload-pre-processing" Oct 30 18:26:25.839785 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.839649 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f5acd91-9868-400b-b97f-f5416477f590" containerName="payload-pre-processing" Oct 30 18:26:25.839785 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.839674 4455 cpu_manager.go:414] "containerMap: removing container" podUID="a3c22942-b988-4208-ba9f-5b8becffa013" containerName="istio-proxy" Oct 30 18:26:25.839785 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.839681 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3c22942-b988-4208-ba9f-5b8becffa013" containerName="istio-proxy" Oct 30 18:26:25.840152 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.839817 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="d14b8e74-04c8-4948-8e26-12c06bc39341" containerName="payload-pre-processing" Oct 30 18:26:25.840152 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.839829 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="c436ce58-c966-4069-bbe3-747b16908679" containerName="payload-processing" Oct 30 18:26:25.840152 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.839843 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="80b01b7a-2c9f-41b5-ba92-9ade95a89cc8" containerName="maas-api" Oct 30 18:26:25.840152 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.839851 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad790c3e-7a8f-462a-835c-870bb8f73002" containerName="istio-proxy" Oct 30 18:26:25.840152 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.839862 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fdeba52-780c-4631-aded-74a165c8c700" containerName="payload-pre-processing" Oct 30 18:26:25.840152 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.839873 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a49c1e9-b602-4d17-828c-a95ea3325c66" containerName="payload-processing" Oct 30 18:26:25.840152 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.839885 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="30592a4b-d20f-44ab-bb1d-7bd5aaf9163d" containerName="payload-processing" Oct 30 18:26:25.840152 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.839895 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f5acd91-9868-400b-b97f-f5416477f590" containerName="payload-pre-processing" Oct 30 18:26:25.840152 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.839907 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3c22942-b988-4208-ba9f-5b8becffa013" containerName="istio-proxy" Oct 30 18:26:25.840152 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.839916 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="13d35525-5ffe-467e-a701-3200ca4facfd" containerName="maas-api" Oct 30 18:26:25.844560 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.844537 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" Oct 30 18:26:25.883268 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.883235 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9"] Oct 30 18:26:25.946822 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.946785 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98345b-38d7-49b3-8092-a41b1f584fb0-credential-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" Oct 30 18:26:25.946993 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.946852 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98345b-38d7-49b3-8092-a41b1f584fb0-workload-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" Oct 30 18:26:25.946993 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.946970 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d98345b-38d7-49b3-8092-a41b1f584fb0-istio-token\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" Oct 30 18:26:25.947100 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.947078 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d98345b-38d7-49b3-8092-a41b1f584fb0-workload-certs\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" Oct 30 18:26:25.947139 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.947109 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d98345b-38d7-49b3-8092-a41b1f584fb0-istio-podinfo\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" Oct 30 18:26:25.947139 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.947129 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d98345b-38d7-49b3-8092-a41b1f584fb0-istio-envoy\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" Oct 30 18:26:25.947204 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.947192 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d98345b-38d7-49b3-8092-a41b1f584fb0-istio-data\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" Oct 30 18:26:25.947306 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.947293 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tghn\" (UniqueName: \"kubernetes.io/projected/1d98345b-38d7-49b3-8092-a41b1f584fb0-kube-api-access-9tghn\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" Oct 30 18:26:25.947384 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:25.947370 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d98345b-38d7-49b3-8092-a41b1f584fb0-istiod-ca-cert\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" Oct 30 18:26:26.048577 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:26.048537 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d98345b-38d7-49b3-8092-a41b1f584fb0-workload-certs\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" Oct 30 18:26:26.048577 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:26.048579 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d98345b-38d7-49b3-8092-a41b1f584fb0-istio-podinfo\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" Oct 30 18:26:26.048818 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:26.048604 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d98345b-38d7-49b3-8092-a41b1f584fb0-istio-envoy\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" Oct 30 18:26:26.048818 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:26.048737 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d98345b-38d7-49b3-8092-a41b1f584fb0-istio-data\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" Oct 30 18:26:26.048933 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:26.048871 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tghn\" (UniqueName: \"kubernetes.io/projected/1d98345b-38d7-49b3-8092-a41b1f584fb0-kube-api-access-9tghn\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" Oct 30 18:26:26.049009 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:26.048974 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d98345b-38d7-49b3-8092-a41b1f584fb0-istiod-ca-cert\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" Oct 30 18:26:26.049009 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:26.048982 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d98345b-38d7-49b3-8092-a41b1f584fb0-workload-certs\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" Oct 30 18:26:26.049122 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:26.049096 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d98345b-38d7-49b3-8092-a41b1f584fb0-istio-data\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" Oct 30 18:26:26.049205 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:26.049180 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98345b-38d7-49b3-8092-a41b1f584fb0-credential-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" Oct 30 18:26:26.049557 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:26.049252 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98345b-38d7-49b3-8092-a41b1f584fb0-workload-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" Oct 30 18:26:26.049557 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:26.049282 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d98345b-38d7-49b3-8092-a41b1f584fb0-istio-token\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" Oct 30 18:26:26.049696 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:26.049594 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98345b-38d7-49b3-8092-a41b1f584fb0-workload-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" Oct 30 18:26:26.049696 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:26.049619 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d98345b-38d7-49b3-8092-a41b1f584fb0-istiod-ca-cert\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" Oct 30 18:26:26.049822 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:26.049797 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98345b-38d7-49b3-8092-a41b1f584fb0-credential-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" Oct 30 18:26:26.051271 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:26.051247 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d98345b-38d7-49b3-8092-a41b1f584fb0-istio-envoy\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" Oct 30 18:26:26.051579 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:26.051557 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d98345b-38d7-49b3-8092-a41b1f584fb0-istio-podinfo\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" Oct 30 18:26:26.079697 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:26.079667 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tghn\" (UniqueName: \"kubernetes.io/projected/1d98345b-38d7-49b3-8092-a41b1f584fb0-kube-api-access-9tghn\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" Oct 30 18:26:26.079844 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:26.079796 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d98345b-38d7-49b3-8092-a41b1f584fb0-istio-token\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" Oct 30 18:26:26.167647 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:26.167559 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" Oct 30 18:26:26.500673 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:26.500637 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9"] Oct 30 18:26:26.501655 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:26:26.501627 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d98345b_38d7_49b3_8092_a41b1f584fb0.slice/crio-dc0a6124b03c57f3bff71f1964ae97e56c1da23ce1622db95e5ce3431c319a8c WatchSource:0}: Error finding container dc0a6124b03c57f3bff71f1964ae97e56c1da23ce1622db95e5ce3431c319a8c: Status 404 returned error can't find the container with id dc0a6124b03c57f3bff71f1964ae97e56c1da23ce1622db95e5ce3431c319a8c Oct 30 18:26:26.504480 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:26.504442 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:26:26.504595 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:26.504528 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:26:26.504595 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:26.504575 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:26:26.822270 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:26.817672 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" event={"ID":"1d98345b-38d7-49b3-8092-a41b1f584fb0","Type":"ContainerStarted","Data":"9d5b84264e6fac78fedbd2349b91b34feda797d9563b0d8e98931812624e7b84"} Oct 30 18:26:26.822270 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:26.817741 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" event={"ID":"1d98345b-38d7-49b3-8092-a41b1f584fb0","Type":"ContainerStarted","Data":"dc0a6124b03c57f3bff71f1964ae97e56c1da23ce1622db95e5ce3431c319a8c"} Oct 30 18:26:27.167936 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:27.167844 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" Oct 30 18:26:27.170087 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:27.170056 4455 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 18:26:27.170240 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:27.170122 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" podUID="1d98345b-38d7-49b3-8092-a41b1f584fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 18:26:28.177349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:28.170721 4455 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 18:26:28.177349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:28.170786 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" podUID="1d98345b-38d7-49b3-8092-a41b1f584fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 18:26:29.182714 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:29.182673 4455 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 18:26:29.183337 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:29.183286 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" podUID="1d98345b-38d7-49b3-8092-a41b1f584fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 18:26:30.185270 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:30.185234 4455 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 18:26:30.192506 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:30.192466 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" podUID="1d98345b-38d7-49b3-8092-a41b1f584fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 18:26:30.425239 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:26:30.425148 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:26:31.182350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:31.170713 4455 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 18:26:31.182350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:31.170791 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" podUID="1d98345b-38d7-49b3-8092-a41b1f584fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 18:26:31.324803 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:31.322945 4455 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-84.ec2.internal" podUID="da45dfd37aee5c345465266021945ad3" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:26:32.174348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:32.173902 4455 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 18:26:32.174348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:32.173964 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" podUID="1d98345b-38d7-49b3-8092-a41b1f584fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 18:26:33.177851 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:33.177803 4455 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 18:26:33.178387 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:33.177880 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" podUID="1d98345b-38d7-49b3-8092-a41b1f584fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 18:26:34.172598 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:34.172563 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" Oct 30 18:26:34.173161 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:34.173134 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" Oct 30 18:26:34.174144 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:34.174123 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" Oct 30 18:26:34.238612 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:34.238554 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" podStartSLOduration=9.238540371 podStartE2EDuration="9.238540371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:26:26.885549257 +0000 UTC m=+2039.373186322" watchObservedRunningTime="2026-10-30 18:26:34.238540371 +0000 UTC m=+2046.726177404" Oct 30 18:26:35.580598 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:35.580565 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-6f78c8fc5f-p5dft"] Oct 30 18:26:35.581460 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:35.581444 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="80b01b7a-2c9f-41b5-ba92-9ade95a89cc8" containerName="maas-api" Oct 30 18:26:35.586514 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:35.586489 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-6f78c8fc5f-p5dft" Oct 30 18:26:35.604162 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:35.604135 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-6f78c8fc5f-p5dft"] Oct 30 18:26:35.618352 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:35.612580 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-602e18-dockercfg-fkk6z\"" Oct 30 18:26:35.618352 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:35.612651 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-602e18\"" Oct 30 18:26:35.752356 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:35.737242 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mb4z\" (UniqueName: \"kubernetes.io/projected/289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5-kube-api-access-4mb4z\") pod \"payload-pre-processing-e2e-ipp-b-602e18-6f78c8fc5f-p5dft\" (UID: \"289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-6f78c8fc5f-p5dft" Oct 30 18:26:35.752903 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:35.752867 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-602e18-6f78c8fc5f-p5dft\" (UID: \"289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-6f78c8fc5f-p5dft" Oct 30 18:26:35.871963 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:35.871739 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mb4z\" (UniqueName: \"kubernetes.io/projected/289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5-kube-api-access-4mb4z\") pod \"payload-pre-processing-e2e-ipp-b-602e18-6f78c8fc5f-p5dft\" (UID: \"289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-6f78c8fc5f-p5dft" Oct 30 18:26:35.873350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:35.872349 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-602e18-6f78c8fc5f-p5dft\" (UID: \"289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-6f78c8fc5f-p5dft" Oct 30 18:26:35.873850 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:35.873829 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-602e18-6f78c8fc5f-p5dft\" (UID: \"289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-6f78c8fc5f-p5dft" Oct 30 18:26:35.981902 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:35.981854 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-602e18-66f64c68db-q5x68"] Oct 30 18:26:35.991617 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:35.991588 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-602e18-66f64c68db-q5x68"] Oct 30 18:26:35.996205 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:35.992591 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-66f64c68db-q5x68" Oct 30 18:26:36.020376 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:36.009390 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mb4z\" (UniqueName: \"kubernetes.io/projected/289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5-kube-api-access-4mb4z\") pod \"payload-pre-processing-e2e-ipp-b-602e18-6f78c8fc5f-p5dft\" (UID: \"289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-6f78c8fc5f-p5dft" Oct 30 18:26:36.076360 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:36.076281 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-602e18-66f64c68db-q5x68\" (UID: \"a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-66f64c68db-q5x68" Oct 30 18:26:36.076566 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:36.076458 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7w2v\" (UniqueName: \"kubernetes.io/projected/a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9-kube-api-access-f7w2v\") pod \"payload-processing-e2e-ipp-b-602e18-66f64c68db-q5x68\" (UID: \"a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-66f64c68db-q5x68" Oct 30 18:26:36.180406 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:36.178139 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-602e18-66f64c68db-q5x68\" (UID: \"a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-66f64c68db-q5x68" Oct 30 18:26:36.180406 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:36.178275 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7w2v\" (UniqueName: \"kubernetes.io/projected/a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9-kube-api-access-f7w2v\") pod \"payload-processing-e2e-ipp-b-602e18-66f64c68db-q5x68\" (UID: \"a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-66f64c68db-q5x68" Oct 30 18:26:36.181346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:36.181294 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-602e18-66f64c68db-q5x68\" (UID: \"a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-66f64c68db-q5x68" Oct 30 18:26:36.207661 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:36.207624 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-6f78c8fc5f-p5dft" Oct 30 18:26:36.209977 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:36.209946 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7w2v\" (UniqueName: \"kubernetes.io/projected/a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9-kube-api-access-f7w2v\") pod \"payload-processing-e2e-ipp-b-602e18-66f64c68db-q5x68\" (UID: \"a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-66f64c68db-q5x68" Oct 30 18:26:36.340638 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:36.340594 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-66f64c68db-q5x68" Oct 30 18:26:36.533264 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:36.532224 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-6f78c8fc5f-p5dft"] Oct 30 18:26:36.784940 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:36.784910 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-602e18-66f64c68db-q5x68"] Oct 30 18:26:36.785574 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:26:36.785546 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c21d3e_9be0_48c6_95ba_1f1d8d186dd9.slice/crio-9468a66262389b855ea685914d78de7fe06543f30842f9db5e906e87d3581cb9 WatchSource:0}: Error finding container 9468a66262389b855ea685914d78de7fe06543f30842f9db5e906e87d3581cb9: Status 404 returned error can't find the container with id 9468a66262389b855ea685914d78de7fe06543f30842f9db5e906e87d3581cb9 Oct 30 18:26:36.994243 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:36.994199 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-6f78c8fc5f-p5dft" event={"ID":"289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5","Type":"ContainerStarted","Data":"2bba86bbf62fd5652b4be287b7d42e5f2526b96c3842acfbb000a5e44c54195d"} Oct 30 18:26:36.994485 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:36.994271 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-6f78c8fc5f-p5dft" event={"ID":"289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5","Type":"ContainerStarted","Data":"71427838ba68e3870cf65abdd6394b55bb122fc3747b66ea5a1868af08f5019d"} Oct 30 18:26:36.994485 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:36.994305 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-6f78c8fc5f-p5dft" Oct 30 18:26:37.002066 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:37.002034 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-66f64c68db-q5x68" event={"ID":"a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9","Type":"ContainerStarted","Data":"8d229e713607a1b4eb29b5c08e4befc2085651183d4d68530966b1e677c48242"} Oct 30 18:26:37.002232 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:37.002093 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-66f64c68db-q5x68" event={"ID":"a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9","Type":"ContainerStarted","Data":"9468a66262389b855ea685914d78de7fe06543f30842f9db5e906e87d3581cb9"} Oct 30 18:26:37.002232 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:37.002155 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-66f64c68db-q5x68" Oct 30 18:26:37.043024 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:37.042935 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-6f78c8fc5f-p5dft" podStartSLOduration=2.042920318 podStartE2EDuration="2.042920318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:26:37.03781466 +0000 UTC m=+2049.525451693" watchObservedRunningTime="2026-10-30 18:26:37.042920318 +0000 UTC m=+2049.530557351" Oct 30 18:26:37.085381 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:37.085303 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-66f64c68db-q5x68" podStartSLOduration=2.085289097 podStartE2EDuration="2.085289097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:26:37.081953323 +0000 UTC m=+2049.569590358" watchObservedRunningTime="2026-10-30 18:26:37.085289097 +0000 UTC m=+2049.572926167" Oct 30 18:26:48.073522 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:48.073490 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-66f64c68db-q5x68" Oct 30 18:26:48.073962 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:48.073938 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-6f78c8fc5f-p5dft" Oct 30 18:26:49.073796 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:49.073763 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs"] Oct 30 18:26:49.080196 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:49.080174 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs" Oct 30 18:26:49.099383 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:49.099354 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-602e18-serving-cert\"" Oct 30 18:26:49.151131 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:49.151098 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs"] Oct 30 18:26:49.261797 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:49.261764 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpk88\" (UniqueName: \"kubernetes.io/projected/b0dbf354-3013-4694-b003-a55b1ddbaea0-kube-api-access-vpk88\") pod \"maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs\" (UID: \"b0dbf354-3013-4694-b003-a55b1ddbaea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs" Oct 30 18:26:49.261984 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:49.261826 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0dbf354-3013-4694-b003-a55b1ddbaea0-metrics-certs\") pod \"maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs\" (UID: \"b0dbf354-3013-4694-b003-a55b1ddbaea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs" Oct 30 18:26:49.262198 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:49.262149 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0dbf354-3013-4694-b003-a55b1ddbaea0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs\" (UID: \"b0dbf354-3013-4694-b003-a55b1ddbaea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs" Oct 30 18:26:49.262452 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:49.262419 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0dbf354-3013-4694-b003-a55b1ddbaea0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs\" (UID: \"b0dbf354-3013-4694-b003-a55b1ddbaea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs" Oct 30 18:26:49.364843 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:49.364735 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0dbf354-3013-4694-b003-a55b1ddbaea0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs\" (UID: \"b0dbf354-3013-4694-b003-a55b1ddbaea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs" Oct 30 18:26:49.365054 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:49.365033 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0dbf354-3013-4694-b003-a55b1ddbaea0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs\" (UID: \"b0dbf354-3013-4694-b003-a55b1ddbaea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs" Oct 30 18:26:49.365239 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:49.365209 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpk88\" (UniqueName: \"kubernetes.io/projected/b0dbf354-3013-4694-b003-a55b1ddbaea0-kube-api-access-vpk88\") pod \"maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs\" (UID: \"b0dbf354-3013-4694-b003-a55b1ddbaea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs" Oct 30 18:26:49.365456 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:49.365282 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0dbf354-3013-4694-b003-a55b1ddbaea0-metrics-certs\") pod \"maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs\" (UID: \"b0dbf354-3013-4694-b003-a55b1ddbaea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs" Oct 30 18:26:49.377404 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:49.369730 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0dbf354-3013-4694-b003-a55b1ddbaea0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs\" (UID: \"b0dbf354-3013-4694-b003-a55b1ddbaea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs" Oct 30 18:26:49.408311 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:49.408256 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0dbf354-3013-4694-b003-a55b1ddbaea0-metrics-certs\") pod \"maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs\" (UID: \"b0dbf354-3013-4694-b003-a55b1ddbaea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs" Oct 30 18:26:49.414044 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:49.413996 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpk88\" (UniqueName: \"kubernetes.io/projected/b0dbf354-3013-4694-b003-a55b1ddbaea0-kube-api-access-vpk88\") pod \"maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs\" (UID: \"b0dbf354-3013-4694-b003-a55b1ddbaea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs" Oct 30 18:26:49.414905 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:49.414877 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0dbf354-3013-4694-b003-a55b1ddbaea0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs\" (UID: \"b0dbf354-3013-4694-b003-a55b1ddbaea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs" Oct 30 18:26:49.699159 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:49.699073 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs" Oct 30 18:26:50.038575 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:50.038534 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs"] Oct 30 18:26:50.043157 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:26:50.043124 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0dbf354_3013_4694_b003_a55b1ddbaea0.slice/crio-7560614abc7406ac6016b9940d204e394464b357f4a8dbb0cebdfc53240cf3db WatchSource:0}: Error finding container 7560614abc7406ac6016b9940d204e394464b357f4a8dbb0cebdfc53240cf3db: Status 404 returned error can't find the container with id 7560614abc7406ac6016b9940d204e394464b357f4a8dbb0cebdfc53240cf3db Oct 30 18:26:50.243876 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:50.243839 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs" event={"ID":"b0dbf354-3013-4694-b003-a55b1ddbaea0","Type":"ContainerStarted","Data":"7560614abc7406ac6016b9940d204e394464b357f4a8dbb0cebdfc53240cf3db"} Oct 30 18:26:51.254294 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:51.254253 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs" event={"ID":"b0dbf354-3013-4694-b003-a55b1ddbaea0","Type":"ContainerStarted","Data":"687ad10b726e1e5ea64de1d35575778fc34cf04bf02a83c808c03a2447357815"} Oct 30 18:26:51.303690 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:51.303629 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs" podStartSLOduration=2.018342541 podStartE2EDuration="2.303613284s" totalImagesPullingTime="285.270743ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:26:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:26:51.301556328 +0000 UTC m=+2063.789193365" watchObservedRunningTime="2026-10-30 18:26:51.303613284 +0000 UTC m=+2063.791250359" Oct 30 18:26:52.262618 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:52.262579 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs" Oct 30 18:26:55.578551 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:55.578496 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45"] Oct 30 18:26:55.579486 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:55.579433 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" podUID="bfbae34d-b472-4dd7-b862-dbb457726886" containerName="maas-api" containerID="cri-o://0176d329f7d90e96028f076ff97a4e57f2d5bff8b91917144636d65cb5d6630b" gracePeriod=30 Oct 30 18:26:56.303486 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:56.302210 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" Oct 30 18:26:56.354660 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:56.354447 4455 generic.go:356] "Generic (PLEG): container finished" podID="bfbae34d-b472-4dd7-b862-dbb457726886" containerID="0176d329f7d90e96028f076ff97a4e57f2d5bff8b91917144636d65cb5d6630b" exitCode=0 Oct 30 18:26:56.354868 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:56.354668 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" event={"ID":"bfbae34d-b472-4dd7-b862-dbb457726886","Type":"ContainerDied","Data":"0176d329f7d90e96028f076ff97a4e57f2d5bff8b91917144636d65cb5d6630b"} Oct 30 18:26:56.354868 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:56.354693 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" Oct 30 18:26:56.354868 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:56.354752 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45" event={"ID":"bfbae34d-b472-4dd7-b862-dbb457726886","Type":"ContainerDied","Data":"ce2154dd0c35efbe17b6e6c1f47b8f6c1001b87b6015026d5698ab73fb230959"} Oct 30 18:26:56.354868 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:56.354783 4455 scope.go:122] "RemoveContainer" containerID="0176d329f7d90e96028f076ff97a4e57f2d5bff8b91917144636d65cb5d6630b" Oct 30 18:26:56.367509 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:56.367484 4455 scope.go:122] "RemoveContainer" containerID="0176d329f7d90e96028f076ff97a4e57f2d5bff8b91917144636d65cb5d6630b" Oct 30 18:26:56.368615 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:26:56.367781 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0176d329f7d90e96028f076ff97a4e57f2d5bff8b91917144636d65cb5d6630b\": container with ID starting with 0176d329f7d90e96028f076ff97a4e57f2d5bff8b91917144636d65cb5d6630b not found: ID does not exist" containerID="0176d329f7d90e96028f076ff97a4e57f2d5bff8b91917144636d65cb5d6630b" Oct 30 18:26:56.368615 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:56.367821 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0176d329f7d90e96028f076ff97a4e57f2d5bff8b91917144636d65cb5d6630b"} err="failed to get container status \"0176d329f7d90e96028f076ff97a4e57f2d5bff8b91917144636d65cb5d6630b\": rpc error: code = NotFound desc = could not find container \"0176d329f7d90e96028f076ff97a4e57f2d5bff8b91917144636d65cb5d6630b\": container with ID starting with 0176d329f7d90e96028f076ff97a4e57f2d5bff8b91917144636d65cb5d6630b not found: ID does not exist" Oct 30 18:26:56.406446 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:56.406416 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfbae34d-b472-4dd7-b862-dbb457726886-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfbae34d-b472-4dd7-b862-dbb457726886-openshift-service-ca\") pod \"bfbae34d-b472-4dd7-b862-dbb457726886\" (UID: \"bfbae34d-b472-4dd7-b862-dbb457726886\") " Oct 30 18:26:56.406629 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:56.406510 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfbae34d-b472-4dd7-b862-dbb457726886-kube-api-access-l4jrn\" (UniqueName: \"kubernetes.io/projected/bfbae34d-b472-4dd7-b862-dbb457726886-kube-api-access-l4jrn\") pod \"bfbae34d-b472-4dd7-b862-dbb457726886\" (UID: \"bfbae34d-b472-4dd7-b862-dbb457726886\") " Oct 30 18:26:56.406696 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:56.406631 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfbae34d-b472-4dd7-b862-dbb457726886-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfbae34d-b472-4dd7-b862-dbb457726886-metrics-certs\") pod \"bfbae34d-b472-4dd7-b862-dbb457726886\" (UID: \"bfbae34d-b472-4dd7-b862-dbb457726886\") " Oct 30 18:26:56.406696 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:56.406678 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfbae34d-b472-4dd7-b862-dbb457726886-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfbae34d-b472-4dd7-b862-dbb457726886-maas-api-tls\") pod \"bfbae34d-b472-4dd7-b862-dbb457726886\" (UID: \"bfbae34d-b472-4dd7-b862-dbb457726886\") " Oct 30 18:26:56.407175 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:56.407145 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfbae34d-b472-4dd7-b862-dbb457726886-openshift-service-ca" pod "bfbae34d-b472-4dd7-b862-dbb457726886" (UID: "bfbae34d-b472-4dd7-b862-dbb457726886"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:56.407301 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:56.407280 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfbae34d-b472-4dd7-b862-dbb457726886-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:26:56.409694 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:56.409583 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfbae34d-b472-4dd7-b862-dbb457726886-metrics-certs" pod "bfbae34d-b472-4dd7-b862-dbb457726886" (UID: "bfbae34d-b472-4dd7-b862-dbb457726886"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:56.410006 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:56.409982 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfbae34d-b472-4dd7-b862-dbb457726886-kube-api-access-l4jrn" pod "bfbae34d-b472-4dd7-b862-dbb457726886" (UID: "bfbae34d-b472-4dd7-b862-dbb457726886"). InnerVolumeSpecName "kube-api-access-l4jrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:56.410295 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:56.410278 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfbae34d-b472-4dd7-b862-dbb457726886-maas-api-tls" pod "bfbae34d-b472-4dd7-b862-dbb457726886" (UID: "bfbae34d-b472-4dd7-b862-dbb457726886"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:56.514362 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:56.508761 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4jrn\" (UniqueName: \"kubernetes.io/projected/bfbae34d-b472-4dd7-b862-dbb457726886-kube-api-access-l4jrn\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:26:56.514362 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:56.508805 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfbae34d-b472-4dd7-b862-dbb457726886-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:26:56.514362 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:56.508817 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfbae34d-b472-4dd7-b862-dbb457726886-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:26:56.631637 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:56.631606 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl"] Oct 30 18:26:56.681812 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:56.681773 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9"] Oct 30 18:26:56.729109 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:56.729018 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45"] Oct 30 18:26:56.733513 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:56.733487 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c2b103-645d98d8df-kkh45"] Oct 30 18:26:57.645286 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:57.645252 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfbae34d-b472-4dd7-b862-dbb457726886" path="/var/lib/kubelet/pods/bfbae34d-b472-4dd7-b862-dbb457726886/volumes" Oct 30 18:26:58.277228 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:58.277199 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs" Oct 30 18:26:58.409831 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:58.409799 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw"] Oct 30 18:26:58.410194 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:58.410143 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw" podUID="da2b820f-4f95-41a7-a438-32c71a5e3679" containerName="maas-api" containerID="cri-o://9ade51f5e34a4b8a162c9a8646b71d79ba0b643050aaabcc3a87aeffbc87e3bc" gracePeriod=30 Oct 30 18:26:59.002362 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:59.001997 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw" Oct 30 18:26:59.143120 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:59.143023 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da2b820f-4f95-41a7-a438-32c71a5e3679-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da2b820f-4f95-41a7-a438-32c71a5e3679-maas-api-tls\") pod \"da2b820f-4f95-41a7-a438-32c71a5e3679\" (UID: \"da2b820f-4f95-41a7-a438-32c71a5e3679\") " Oct 30 18:26:59.143120 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:59.143115 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da2b820f-4f95-41a7-a438-32c71a5e3679-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da2b820f-4f95-41a7-a438-32c71a5e3679-openshift-service-ca\") pod \"da2b820f-4f95-41a7-a438-32c71a5e3679\" (UID: \"da2b820f-4f95-41a7-a438-32c71a5e3679\") " Oct 30 18:26:59.143385 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:59.143198 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da2b820f-4f95-41a7-a438-32c71a5e3679-kube-api-access-wrx2t\" (UniqueName: \"kubernetes.io/projected/da2b820f-4f95-41a7-a438-32c71a5e3679-kube-api-access-wrx2t\") pod \"da2b820f-4f95-41a7-a438-32c71a5e3679\" (UID: \"da2b820f-4f95-41a7-a438-32c71a5e3679\") " Oct 30 18:26:59.143385 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:59.143229 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da2b820f-4f95-41a7-a438-32c71a5e3679-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da2b820f-4f95-41a7-a438-32c71a5e3679-metrics-certs\") pod \"da2b820f-4f95-41a7-a438-32c71a5e3679\" (UID: \"da2b820f-4f95-41a7-a438-32c71a5e3679\") " Oct 30 18:26:59.143555 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:59.143528 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da2b820f-4f95-41a7-a438-32c71a5e3679-openshift-service-ca" pod "da2b820f-4f95-41a7-a438-32c71a5e3679" (UID: "da2b820f-4f95-41a7-a438-32c71a5e3679"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:59.144067 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:59.144040 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da2b820f-4f95-41a7-a438-32c71a5e3679-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:26:59.145623 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:59.145595 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da2b820f-4f95-41a7-a438-32c71a5e3679-kube-api-access-wrx2t" pod "da2b820f-4f95-41a7-a438-32c71a5e3679" (UID: "da2b820f-4f95-41a7-a438-32c71a5e3679"). InnerVolumeSpecName "kube-api-access-wrx2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:59.145722 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:59.145677 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da2b820f-4f95-41a7-a438-32c71a5e3679-metrics-certs" pod "da2b820f-4f95-41a7-a438-32c71a5e3679" (UID: "da2b820f-4f95-41a7-a438-32c71a5e3679"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:59.145722 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:59.145688 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da2b820f-4f95-41a7-a438-32c71a5e3679-maas-api-tls" pod "da2b820f-4f95-41a7-a438-32c71a5e3679" (UID: "da2b820f-4f95-41a7-a438-32c71a5e3679"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:59.245719 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:59.245683 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da2b820f-4f95-41a7-a438-32c71a5e3679-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:26:59.245719 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:59.245711 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrx2t\" (UniqueName: \"kubernetes.io/projected/da2b820f-4f95-41a7-a438-32c71a5e3679-kube-api-access-wrx2t\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:26:59.245719 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:59.245721 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da2b820f-4f95-41a7-a438-32c71a5e3679-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:26:59.399398 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:59.396758 4455 generic.go:356] "Generic (PLEG): container finished" podID="da2b820f-4f95-41a7-a438-32c71a5e3679" containerID="9ade51f5e34a4b8a162c9a8646b71d79ba0b643050aaabcc3a87aeffbc87e3bc" exitCode=0 Oct 30 18:26:59.399398 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:59.396896 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw" event={"ID":"da2b820f-4f95-41a7-a438-32c71a5e3679","Type":"ContainerDied","Data":"9ade51f5e34a4b8a162c9a8646b71d79ba0b643050aaabcc3a87aeffbc87e3bc"} Oct 30 18:26:59.399398 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:59.396950 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw" event={"ID":"da2b820f-4f95-41a7-a438-32c71a5e3679","Type":"ContainerDied","Data":"4523de5fc36a60071c94eaaf84035ed60918b355f3348c480e44de6cce368f8b"} Oct 30 18:26:59.399398 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:59.396979 4455 scope.go:122] "RemoveContainer" containerID="9ade51f5e34a4b8a162c9a8646b71d79ba0b643050aaabcc3a87aeffbc87e3bc" Oct 30 18:26:59.399398 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:59.397100 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw" Oct 30 18:26:59.416103 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:59.415409 4455 scope.go:122] "RemoveContainer" containerID="9ade51f5e34a4b8a162c9a8646b71d79ba0b643050aaabcc3a87aeffbc87e3bc" Oct 30 18:26:59.416934 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:26:59.416903 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ade51f5e34a4b8a162c9a8646b71d79ba0b643050aaabcc3a87aeffbc87e3bc\": container with ID starting with 9ade51f5e34a4b8a162c9a8646b71d79ba0b643050aaabcc3a87aeffbc87e3bc not found: ID does not exist" containerID="9ade51f5e34a4b8a162c9a8646b71d79ba0b643050aaabcc3a87aeffbc87e3bc" Oct 30 18:26:59.417021 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:59.416946 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ade51f5e34a4b8a162c9a8646b71d79ba0b643050aaabcc3a87aeffbc87e3bc"} err="failed to get container status \"9ade51f5e34a4b8a162c9a8646b71d79ba0b643050aaabcc3a87aeffbc87e3bc\": rpc error: code = NotFound desc = could not find container \"9ade51f5e34a4b8a162c9a8646b71d79ba0b643050aaabcc3a87aeffbc87e3bc\": container with ID starting with 9ade51f5e34a4b8a162c9a8646b71d79ba0b643050aaabcc3a87aeffbc87e3bc not found: ID does not exist" Oct 30 18:26:59.466348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:59.466299 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw"] Oct 30 18:26:59.481861 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:59.481830 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-42fad7-64b7454f8-5m2mw"] Oct 30 18:26:59.635875 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:26:59.635845 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da2b820f-4f95-41a7-a438-32c71a5e3679" path="/var/lib/kubelet/pods/da2b820f-4f95-41a7-a438-32c71a5e3679/volumes" Oct 30 18:27:00.387434 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:27:00.387396 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:27:01.632651 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:01.632604 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl" podUID="6a166721-a635-4df9-adfe-7887eb5e7b39" containerName="payload-processing" containerID="cri-o://2971e893381552fe53bfd096f7337a5df321e7c50a127b113243cb274e212b23" gracePeriod=25 Oct 30 18:27:01.683153 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:01.683087 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9" podUID="46eee9bc-87c0-4421-92e7-547a494dbf3c" containerName="payload-pre-processing" containerID="cri-o://80818c409c04de2cd7a34df5600a5a2b271fb0fd8b404ae5d2969a81297e70c4" gracePeriod=25 Oct 30 18:27:01.998965 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:01.998933 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl" Oct 30 18:27:02.006956 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:02.006380 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a166721-a635-4df9-adfe-7887eb5e7b39-kube-api-access-spq86\" (UniqueName: \"kubernetes.io/projected/6a166721-a635-4df9-adfe-7887eb5e7b39-kube-api-access-spq86\") pod \"6a166721-a635-4df9-adfe-7887eb5e7b39\" (UID: \"6a166721-a635-4df9-adfe-7887eb5e7b39\") " Oct 30 18:27:02.006956 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:02.006513 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a166721-a635-4df9-adfe-7887eb5e7b39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a166721-a635-4df9-adfe-7887eb5e7b39-plugins-config-volume\") pod \"6a166721-a635-4df9-adfe-7887eb5e7b39\" (UID: \"6a166721-a635-4df9-adfe-7887eb5e7b39\") " Oct 30 18:27:02.007383 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:02.007351 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a166721-a635-4df9-adfe-7887eb5e7b39-plugins-config-volume" pod "6a166721-a635-4df9-adfe-7887eb5e7b39" (UID: "6a166721-a635-4df9-adfe-7887eb5e7b39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:02.011131 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:02.009953 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a166721-a635-4df9-adfe-7887eb5e7b39-kube-api-access-spq86" pod "6a166721-a635-4df9-adfe-7887eb5e7b39" (UID: "6a166721-a635-4df9-adfe-7887eb5e7b39"). InnerVolumeSpecName "kube-api-access-spq86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:02.038596 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:02.038574 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9" Oct 30 18:27:02.111345 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:02.111008 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46eee9bc-87c0-4421-92e7-547a494dbf3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46eee9bc-87c0-4421-92e7-547a494dbf3c-plugins-config-volume\") pod \"46eee9bc-87c0-4421-92e7-547a494dbf3c\" (UID: \"46eee9bc-87c0-4421-92e7-547a494dbf3c\") " Oct 30 18:27:02.111820 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:02.111723 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46eee9bc-87c0-4421-92e7-547a494dbf3c-plugins-config-volume" pod "46eee9bc-87c0-4421-92e7-547a494dbf3c" (UID: "46eee9bc-87c0-4421-92e7-547a494dbf3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:02.111923 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:02.111907 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46eee9bc-87c0-4421-92e7-547a494dbf3c-kube-api-access-4qb4k\" (UniqueName: \"kubernetes.io/projected/46eee9bc-87c0-4421-92e7-547a494dbf3c-kube-api-access-4qb4k\") pod \"46eee9bc-87c0-4421-92e7-547a494dbf3c\" (UID: \"46eee9bc-87c0-4421-92e7-547a494dbf3c\") " Oct 30 18:27:02.112931 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:02.112905 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spq86\" (UniqueName: \"kubernetes.io/projected/6a166721-a635-4df9-adfe-7887eb5e7b39-kube-api-access-spq86\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:27:02.113095 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:02.112935 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a166721-a635-4df9-adfe-7887eb5e7b39-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:27:02.113095 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:02.112949 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46eee9bc-87c0-4421-92e7-547a494dbf3c-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:27:02.115018 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:02.114962 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46eee9bc-87c0-4421-92e7-547a494dbf3c-kube-api-access-4qb4k" pod "46eee9bc-87c0-4421-92e7-547a494dbf3c" (UID: "46eee9bc-87c0-4421-92e7-547a494dbf3c"). InnerVolumeSpecName "kube-api-access-4qb4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:02.214678 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:02.214591 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qb4k\" (UniqueName: \"kubernetes.io/projected/46eee9bc-87c0-4421-92e7-547a494dbf3c-kube-api-access-4qb4k\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:27:02.427143 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:02.427106 4455 generic.go:356] "Generic (PLEG): container finished" podID="46eee9bc-87c0-4421-92e7-547a494dbf3c" containerID="80818c409c04de2cd7a34df5600a5a2b271fb0fd8b404ae5d2969a81297e70c4" exitCode=0 Oct 30 18:27:02.427333 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:02.427157 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9" Oct 30 18:27:02.427333 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:02.427196 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9" event={"ID":"46eee9bc-87c0-4421-92e7-547a494dbf3c","Type":"ContainerDied","Data":"80818c409c04de2cd7a34df5600a5a2b271fb0fd8b404ae5d2969a81297e70c4"} Oct 30 18:27:02.427333 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:02.427248 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9" event={"ID":"46eee9bc-87c0-4421-92e7-547a494dbf3c","Type":"ContainerDied","Data":"b7251e7e86241eaef63d771241c80ef18494023ae25fd7f7a3b5f2dae082b2c0"} Oct 30 18:27:02.427333 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:02.427271 4455 scope.go:122] "RemoveContainer" containerID="80818c409c04de2cd7a34df5600a5a2b271fb0fd8b404ae5d2969a81297e70c4" Oct 30 18:27:02.429038 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:02.428909 4455 generic.go:356] "Generic (PLEG): container finished" podID="6a166721-a635-4df9-adfe-7887eb5e7b39" containerID="2971e893381552fe53bfd096f7337a5df321e7c50a127b113243cb274e212b23" exitCode=0 Oct 30 18:27:02.429038 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:02.428989 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl" event={"ID":"6a166721-a635-4df9-adfe-7887eb5e7b39","Type":"ContainerDied","Data":"2971e893381552fe53bfd096f7337a5df321e7c50a127b113243cb274e212b23"} Oct 30 18:27:02.429038 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:02.429031 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl" event={"ID":"6a166721-a635-4df9-adfe-7887eb5e7b39","Type":"ContainerDied","Data":"26990acb9af6b5fa533fdfab98834bd9a48762c22925799fc8396ef6feeccaee"} Oct 30 18:27:02.429442 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:02.429417 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl" Oct 30 18:27:02.440516 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:02.440494 4455 scope.go:122] "RemoveContainer" containerID="80818c409c04de2cd7a34df5600a5a2b271fb0fd8b404ae5d2969a81297e70c4" Oct 30 18:27:02.440860 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:27:02.440840 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80818c409c04de2cd7a34df5600a5a2b271fb0fd8b404ae5d2969a81297e70c4\": container with ID starting with 80818c409c04de2cd7a34df5600a5a2b271fb0fd8b404ae5d2969a81297e70c4 not found: ID does not exist" containerID="80818c409c04de2cd7a34df5600a5a2b271fb0fd8b404ae5d2969a81297e70c4" Oct 30 18:27:02.440959 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:02.440868 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80818c409c04de2cd7a34df5600a5a2b271fb0fd8b404ae5d2969a81297e70c4"} err="failed to get container status \"80818c409c04de2cd7a34df5600a5a2b271fb0fd8b404ae5d2969a81297e70c4\": rpc error: code = NotFound desc = could not find container \"80818c409c04de2cd7a34df5600a5a2b271fb0fd8b404ae5d2969a81297e70c4\": container with ID starting with 80818c409c04de2cd7a34df5600a5a2b271fb0fd8b404ae5d2969a81297e70c4 not found: ID does not exist" Oct 30 18:27:02.440959 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:02.440886 4455 scope.go:122] "RemoveContainer" containerID="2971e893381552fe53bfd096f7337a5df321e7c50a127b113243cb274e212b23" Oct 30 18:27:02.451451 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:02.451426 4455 scope.go:122] "RemoveContainer" containerID="2971e893381552fe53bfd096f7337a5df321e7c50a127b113243cb274e212b23" Oct 30 18:27:02.451740 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:27:02.451720 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2971e893381552fe53bfd096f7337a5df321e7c50a127b113243cb274e212b23\": container with ID starting with 2971e893381552fe53bfd096f7337a5df321e7c50a127b113243cb274e212b23 not found: ID does not exist" containerID="2971e893381552fe53bfd096f7337a5df321e7c50a127b113243cb274e212b23" Oct 30 18:27:02.451831 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:02.451749 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2971e893381552fe53bfd096f7337a5df321e7c50a127b113243cb274e212b23"} err="failed to get container status \"2971e893381552fe53bfd096f7337a5df321e7c50a127b113243cb274e212b23\": rpc error: code = NotFound desc = could not find container \"2971e893381552fe53bfd096f7337a5df321e7c50a127b113243cb274e212b23\": container with ID starting with 2971e893381552fe53bfd096f7337a5df321e7c50a127b113243cb274e212b23 not found: ID does not exist" Oct 30 18:27:02.486903 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:02.486870 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9"] Oct 30 18:27:02.500158 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:02.500130 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-42fad7-8495f45cf-md2x9"] Oct 30 18:27:02.528311 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:02.528277 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl"] Oct 30 18:27:02.532792 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:02.532764 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-42fad7-78f7c686db-wxcwl"] Oct 30 18:27:03.689876 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:03.689837 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46eee9bc-87c0-4421-92e7-547a494dbf3c" path="/var/lib/kubelet/pods/46eee9bc-87c0-4421-92e7-547a494dbf3c/volumes" Oct 30 18:27:03.691258 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:03.691234 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a166721-a635-4df9-adfe-7887eb5e7b39" path="/var/lib/kubelet/pods/6a166721-a635-4df9-adfe-7887eb5e7b39/volumes" Oct 30 18:27:04.471349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:04.454846 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx"] Oct 30 18:27:04.475398 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:04.472079 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" podUID="45acf61c-885f-4dc5-9ca3-d8293675107a" containerName="istio-proxy" containerID="cri-o://a9f84fa6101c8e795a8d446b5021ccb549fd9517952f2308bbba9fe16fb641e3" gracePeriod=30 Oct 30 18:27:04.670347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:04.669710 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk"] Oct 30 18:27:04.670347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:04.670025 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" podUID="cb055040-07c9-46eb-8649-dd1db076ee59" containerName="istio-proxy" containerID="cri-o://0d334a0273c56ccdca0232e0f97b01f554b5972b51a90a1003fecf9847490c6d" gracePeriod=30 Oct 30 18:27:09.400535 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:09.387427 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:09.446383 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:27:09.409683 4455 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.754s" Oct 30 18:27:09.878112 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:09.877823 4455 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:27:09.878112 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:09.877911 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" podUID="d2c2d808-5274-484e-8520-1f88fa79fc47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:10.590408 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:10.583891 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:27:10.623226 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:10.597496 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:27:10.981516 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:10.959950 4455 generic.go:356] "Generic (PLEG): container finished" podID="45acf61c-885f-4dc5-9ca3-d8293675107a" containerID="a9f84fa6101c8e795a8d446b5021ccb549fd9517952f2308bbba9fe16fb641e3" exitCode=0 Oct 30 18:27:10.982088 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:10.982051 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" event={"ID":"45acf61c-885f-4dc5-9ca3-d8293675107a","Type":"ContainerDied","Data":"a9f84fa6101c8e795a8d446b5021ccb549fd9517952f2308bbba9fe16fb641e3"} Oct 30 18:27:11.042927 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:11.041979 4455 generic.go:356] "Generic (PLEG): container finished" podID="cb055040-07c9-46eb-8649-dd1db076ee59" containerID="0d334a0273c56ccdca0232e0f97b01f554b5972b51a90a1003fecf9847490c6d" exitCode=0 Oct 30 18:27:11.042927 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:11.042594 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" event={"ID":"cb055040-07c9-46eb-8649-dd1db076ee59","Type":"ContainerDied","Data":"0d334a0273c56ccdca0232e0f97b01f554b5972b51a90a1003fecf9847490c6d"} Oct 30 18:27:11.667221 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:11.640441 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:11.860368 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:11.844910 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-602e18-66f64c68db-q5x68"] Oct 30 18:27:12.015520 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:12.006777 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-6f78c8fc5f-p5dft"] Oct 30 18:27:12.678351 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:12.651806 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:13.647113 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:13.647037 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" event={"ID":"45acf61c-885f-4dc5-9ca3-d8293675107a","Type":"ContainerDied","Data":"a845733dc16e3b91a04c22b8ebc9d72fe59f4089c5fb7942b8162ca332a87d25"} Oct 30 18:27:13.746374 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:13.708832 4455 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a845733dc16e3b91a04c22b8ebc9d72fe59f4089c5fb7942b8162ca332a87d25" Oct 30 18:27:13.960409 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:13.730452 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:14.057056 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:13.757701 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" Oct 30 18:27:14.176098 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.176057 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:27:14.176421 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.176392 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" podUID="4e7b0c73-0041-4549-b47e-76a7aaed2b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:14.219407 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.209020 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45acf61c-885f-4dc5-9ca3-d8293675107a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45acf61c-885f-4dc5-9ca3-d8293675107a-istio-podinfo\") pod \"45acf61c-885f-4dc5-9ca3-d8293675107a\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " Oct 30 18:27:14.219785 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.219752 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45acf61c-885f-4dc5-9ca3-d8293675107a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45acf61c-885f-4dc5-9ca3-d8293675107a-credential-socket\") pod \"45acf61c-885f-4dc5-9ca3-d8293675107a\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " Oct 30 18:27:14.225499 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.225465 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45acf61c-885f-4dc5-9ca3-d8293675107a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45acf61c-885f-4dc5-9ca3-d8293675107a-istiod-ca-cert\") pod \"45acf61c-885f-4dc5-9ca3-d8293675107a\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " Oct 30 18:27:14.235205 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.225751 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45acf61c-885f-4dc5-9ca3-d8293675107a-istio-token\" (UniqueName: \"kubernetes.io/projected/45acf61c-885f-4dc5-9ca3-d8293675107a-istio-token\") pod \"45acf61c-885f-4dc5-9ca3-d8293675107a\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " Oct 30 18:27:14.261906 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.248227 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45acf61c-885f-4dc5-9ca3-d8293675107a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45acf61c-885f-4dc5-9ca3-d8293675107a-workload-certs\") pod \"45acf61c-885f-4dc5-9ca3-d8293675107a\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " Oct 30 18:27:14.261906 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.248279 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45acf61c-885f-4dc5-9ca3-d8293675107a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45acf61c-885f-4dc5-9ca3-d8293675107a-istio-data\") pod \"45acf61c-885f-4dc5-9ca3-d8293675107a\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " Oct 30 18:27:14.261906 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.248380 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45acf61c-885f-4dc5-9ca3-d8293675107a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45acf61c-885f-4dc5-9ca3-d8293675107a-istio-envoy\") pod \"45acf61c-885f-4dc5-9ca3-d8293675107a\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " Oct 30 18:27:14.261906 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.248490 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45acf61c-885f-4dc5-9ca3-d8293675107a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45acf61c-885f-4dc5-9ca3-d8293675107a-workload-socket\") pod \"45acf61c-885f-4dc5-9ca3-d8293675107a\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " Oct 30 18:27:14.261906 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.248619 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45acf61c-885f-4dc5-9ca3-d8293675107a-kube-api-access-wsfss\" (UniqueName: \"kubernetes.io/projected/45acf61c-885f-4dc5-9ca3-d8293675107a-kube-api-access-wsfss\") pod \"45acf61c-885f-4dc5-9ca3-d8293675107a\" (UID: \"45acf61c-885f-4dc5-9ca3-d8293675107a\") " Oct 30 18:27:14.289803 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.289764 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45acf61c-885f-4dc5-9ca3-d8293675107a-workload-certs" pod "45acf61c-885f-4dc5-9ca3-d8293675107a" (UID: "45acf61c-885f-4dc5-9ca3-d8293675107a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:14.293401 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.293368 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45acf61c-885f-4dc5-9ca3-d8293675107a-istio-data" pod "45acf61c-885f-4dc5-9ca3-d8293675107a" (UID: "45acf61c-885f-4dc5-9ca3-d8293675107a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:14.418923 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.418887 4455 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45acf61c-885f-4dc5-9ca3-d8293675107a-workload-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:27:14.423404 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.423378 4455 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45acf61c-885f-4dc5-9ca3-d8293675107a-istio-data\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:27:14.423828 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.423806 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45acf61c-885f-4dc5-9ca3-d8293675107a-workload-socket" pod "45acf61c-885f-4dc5-9ca3-d8293675107a" (UID: "45acf61c-885f-4dc5-9ca3-d8293675107a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:14.424082 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.424056 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45acf61c-885f-4dc5-9ca3-d8293675107a-kube-api-access-wsfss" pod "45acf61c-885f-4dc5-9ca3-d8293675107a" (UID: "45acf61c-885f-4dc5-9ca3-d8293675107a"). InnerVolumeSpecName "kube-api-access-wsfss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:14.598563 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.434229 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45acf61c-885f-4dc5-9ca3-d8293675107a-istiod-ca-cert" pod "45acf61c-885f-4dc5-9ca3-d8293675107a" (UID: "45acf61c-885f-4dc5-9ca3-d8293675107a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:14.598563 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.529895 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" Oct 30 18:27:14.714306 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.714265 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45acf61c-885f-4dc5-9ca3-d8293675107a-istio-podinfo" pod "45acf61c-885f-4dc5-9ca3-d8293675107a" (UID: "45acf61c-885f-4dc5-9ca3-d8293675107a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:27:14.714633 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.714598 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45acf61c-885f-4dc5-9ca3-d8293675107a-credential-socket" pod "45acf61c-885f-4dc5-9ca3-d8293675107a" (UID: "45acf61c-885f-4dc5-9ca3-d8293675107a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:14.728352 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.718760 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45acf61c-885f-4dc5-9ca3-d8293675107a-istio-envoy" pod "45acf61c-885f-4dc5-9ca3-d8293675107a" (UID: "45acf61c-885f-4dc5-9ca3-d8293675107a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:14.728352 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.718921 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45acf61c-885f-4dc5-9ca3-d8293675107a-istio-token" pod "45acf61c-885f-4dc5-9ca3-d8293675107a" (UID: "45acf61c-885f-4dc5-9ca3-d8293675107a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:14.759479 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.747826 4455 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45acf61c-885f-4dc5-9ca3-d8293675107a-workload-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:27:14.759479 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.747942 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsfss\" (UniqueName: \"kubernetes.io/projected/45acf61c-885f-4dc5-9ca3-d8293675107a-kube-api-access-wsfss\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:27:14.759479 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.747991 4455 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45acf61c-885f-4dc5-9ca3-d8293675107a-istio-podinfo\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:27:14.759479 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.748013 4455 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45acf61c-885f-4dc5-9ca3-d8293675107a-credential-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:27:14.759479 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.748059 4455 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45acf61c-885f-4dc5-9ca3-d8293675107a-istiod-ca-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:27:14.759479 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.748074 4455 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45acf61c-885f-4dc5-9ca3-d8293675107a-istio-token\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:27:14.759479 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.748087 4455 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45acf61c-885f-4dc5-9ca3-d8293675107a-istio-envoy\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:27:14.864404 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.858999 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:27:14.864943 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.864902 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb055040-07c9-46eb-8649-dd1db076ee59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb055040-07c9-46eb-8649-dd1db076ee59-credential-socket\") pod \"cb055040-07c9-46eb-8649-dd1db076ee59\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " Oct 30 18:27:14.865456 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.865432 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb055040-07c9-46eb-8649-dd1db076ee59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb055040-07c9-46eb-8649-dd1db076ee59-workload-socket\") pod \"cb055040-07c9-46eb-8649-dd1db076ee59\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " Oct 30 18:27:14.865653 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.865634 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb055040-07c9-46eb-8649-dd1db076ee59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb055040-07c9-46eb-8649-dd1db076ee59-istio-envoy\") pod \"cb055040-07c9-46eb-8649-dd1db076ee59\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " Oct 30 18:27:14.866099 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.866079 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb055040-07c9-46eb-8649-dd1db076ee59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb055040-07c9-46eb-8649-dd1db076ee59-workload-certs\") pod \"cb055040-07c9-46eb-8649-dd1db076ee59\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " Oct 30 18:27:14.866253 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.866236 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb055040-07c9-46eb-8649-dd1db076ee59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb055040-07c9-46eb-8649-dd1db076ee59-istiod-ca-cert\") pod \"cb055040-07c9-46eb-8649-dd1db076ee59\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " Oct 30 18:27:14.866485 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.866463 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb055040-07c9-46eb-8649-dd1db076ee59-istio-token\" (UniqueName: \"kubernetes.io/projected/cb055040-07c9-46eb-8649-dd1db076ee59-istio-token\") pod \"cb055040-07c9-46eb-8649-dd1db076ee59\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " Oct 30 18:27:14.866992 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.866972 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb055040-07c9-46eb-8649-dd1db076ee59-kube-api-access-cx6vl\" (UniqueName: \"kubernetes.io/projected/cb055040-07c9-46eb-8649-dd1db076ee59-kube-api-access-cx6vl\") pod \"cb055040-07c9-46eb-8649-dd1db076ee59\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " Oct 30 18:27:14.867677 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.867657 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb055040-07c9-46eb-8649-dd1db076ee59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb055040-07c9-46eb-8649-dd1db076ee59-istio-data\") pod \"cb055040-07c9-46eb-8649-dd1db076ee59\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " Oct 30 18:27:14.886483 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.886443 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb055040-07c9-46eb-8649-dd1db076ee59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb055040-07c9-46eb-8649-dd1db076ee59-istio-podinfo\") pod \"cb055040-07c9-46eb-8649-dd1db076ee59\" (UID: \"cb055040-07c9-46eb-8649-dd1db076ee59\") " Oct 30 18:27:14.886912 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.865805 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb055040-07c9-46eb-8649-dd1db076ee59-credential-socket" pod "cb055040-07c9-46eb-8649-dd1db076ee59" (UID: "cb055040-07c9-46eb-8649-dd1db076ee59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:14.901722 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.866631 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb055040-07c9-46eb-8649-dd1db076ee59-workload-socket" pod "cb055040-07c9-46eb-8649-dd1db076ee59" (UID: "cb055040-07c9-46eb-8649-dd1db076ee59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:14.901722 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.866902 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb055040-07c9-46eb-8649-dd1db076ee59-workload-certs" pod "cb055040-07c9-46eb-8649-dd1db076ee59" (UID: "cb055040-07c9-46eb-8649-dd1db076ee59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:14.901722 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.867257 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb055040-07c9-46eb-8649-dd1db076ee59-istiod-ca-cert" pod "cb055040-07c9-46eb-8649-dd1db076ee59" (UID: "cb055040-07c9-46eb-8649-dd1db076ee59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:14.901722 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.869073 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb055040-07c9-46eb-8649-dd1db076ee59-istio-data" pod "cb055040-07c9-46eb-8649-dd1db076ee59" (UID: "cb055040-07c9-46eb-8649-dd1db076ee59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:14.902091 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.900759 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb055040-07c9-46eb-8649-dd1db076ee59-kube-api-access-cx6vl" pod "cb055040-07c9-46eb-8649-dd1db076ee59" (UID: "cb055040-07c9-46eb-8649-dd1db076ee59"). InnerVolumeSpecName "kube-api-access-cx6vl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:14.928919 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.924686 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb055040-07c9-46eb-8649-dd1db076ee59-istio-envoy" pod "cb055040-07c9-46eb-8649-dd1db076ee59" (UID: "cb055040-07c9-46eb-8649-dd1db076ee59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:14.971216 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.970968 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx" Oct 30 18:27:14.985446 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.985420 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" Oct 30 18:27:14.985946 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.985912 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk" event={"ID":"cb055040-07c9-46eb-8649-dd1db076ee59","Type":"ContainerDied","Data":"42a211f08fe3e0f8cd779102c3854e1178663069fc5b776bc3132144c96eb265"} Oct 30 18:27:14.986097 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.986085 4455 scope.go:122] "RemoveContainer" containerID="0d334a0273c56ccdca0232e0f97b01f554b5972b51a90a1003fecf9847490c6d" Oct 30 18:27:14.989969 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.989938 4455 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb055040-07c9-46eb-8649-dd1db076ee59-credential-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:27:14.990085 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.989977 4455 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb055040-07c9-46eb-8649-dd1db076ee59-workload-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:27:14.990085 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.989994 4455 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb055040-07c9-46eb-8649-dd1db076ee59-istio-envoy\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:27:14.990085 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.990008 4455 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb055040-07c9-46eb-8649-dd1db076ee59-workload-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:27:14.990085 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.990021 4455 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb055040-07c9-46eb-8649-dd1db076ee59-istiod-ca-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:27:14.990085 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.990034 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx6vl\" (UniqueName: \"kubernetes.io/projected/cb055040-07c9-46eb-8649-dd1db076ee59-kube-api-access-cx6vl\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:27:14.990085 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:14.990048 4455 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb055040-07c9-46eb-8649-dd1db076ee59-istio-data\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:27:15.047545 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:15.047503 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb055040-07c9-46eb-8649-dd1db076ee59-istio-podinfo" pod "cb055040-07c9-46eb-8649-dd1db076ee59" (UID: "cb055040-07c9-46eb-8649-dd1db076ee59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:27:15.060636 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:15.060593 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb055040-07c9-46eb-8649-dd1db076ee59-istio-token" pod "cb055040-07c9-46eb-8649-dd1db076ee59" (UID: "cb055040-07c9-46eb-8649-dd1db076ee59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:15.094372 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:15.094315 4455 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb055040-07c9-46eb-8649-dd1db076ee59-istio-podinfo\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:27:15.094655 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:15.094637 4455 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb055040-07c9-46eb-8649-dd1db076ee59-istio-token\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:27:15.281456 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:15.281410 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx"] Oct 30 18:27:15.375593 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:15.375550 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-knfgx"] Oct 30 18:27:15.516606 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:15.516569 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk"] Oct 30 18:27:15.523785 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:15.523750 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-xbknk"] Oct 30 18:27:15.637855 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:15.637811 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45acf61c-885f-4dc5-9ca3-d8293675107a" path="/var/lib/kubelet/pods/45acf61c-885f-4dc5-9ca3-d8293675107a/volumes" Oct 30 18:27:15.638585 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:15.638559 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb055040-07c9-46eb-8649-dd1db076ee59" path="/var/lib/kubelet/pods/cb055040-07c9-46eb-8649-dd1db076ee59/volumes" Oct 30 18:27:16.845884 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:16.845840 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-66f64c68db-q5x68" podUID="a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9" containerName="payload-processing" containerID="cri-o://8d229e713607a1b4eb29b5c08e4befc2085651183d4d68530966b1e677c48242" gracePeriod=25 Oct 30 18:27:16.992396 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:16.992316 4455 generic.go:356] "Generic (PLEG): container finished" podID="a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9" containerID="8d229e713607a1b4eb29b5c08e4befc2085651183d4d68530966b1e677c48242" exitCode=0 Oct 30 18:27:16.992553 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:16.992410 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-66f64c68db-q5x68" event={"ID":"a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9","Type":"ContainerDied","Data":"8d229e713607a1b4eb29b5c08e4befc2085651183d4d68530966b1e677c48242"} Oct 30 18:27:17.007501 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:17.007453 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-6f78c8fc5f-p5dft" podUID="289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5" containerName="payload-pre-processing" containerID="cri-o://2bba86bbf62fd5652b4be287b7d42e5f2526b96c3842acfbb000a5e44c54195d" gracePeriod=25 Oct 30 18:27:17.146436 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:17.146408 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-66f64c68db-q5x68" Oct 30 18:27:17.261782 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:17.261738 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9-kube-api-access-f7w2v\" (UniqueName: \"kubernetes.io/projected/a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9-kube-api-access-f7w2v\") pod \"a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9\" (UID: \"a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9\") " Oct 30 18:27:17.262155 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:17.262115 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9-plugins-config-volume\") pod \"a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9\" (UID: \"a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9\") " Oct 30 18:27:17.263747 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:17.263705 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9-plugins-config-volume" pod "a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9" (UID: "a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:17.283128 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:17.283051 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9-kube-api-access-f7w2v" pod "a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9" (UID: "a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9"). InnerVolumeSpecName "kube-api-access-f7w2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:17.358605 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:17.358567 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-6f78c8fc5f-p5dft" Oct 30 18:27:17.369256 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:17.369113 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7w2v\" (UniqueName: \"kubernetes.io/projected/a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9-kube-api-access-f7w2v\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:27:17.369256 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:17.369155 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:27:17.470105 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:17.470057 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5-plugins-config-volume\") pod \"289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5\" (UID: \"289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5\") " Oct 30 18:27:17.470268 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:17.470120 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5-kube-api-access-4mb4z\" (UniqueName: \"kubernetes.io/projected/289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5-kube-api-access-4mb4z\") pod \"289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5\" (UID: \"289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5\") " Oct 30 18:27:17.470645 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:17.470613 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5-plugins-config-volume" pod "289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5" (UID: "289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:17.470858 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:17.470833 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:27:17.472921 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:17.472892 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5-kube-api-access-4mb4z" pod "289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5" (UID: "289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5"). InnerVolumeSpecName "kube-api-access-4mb4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:17.572775 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:17.572739 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mb4z\" (UniqueName: \"kubernetes.io/projected/289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5-kube-api-access-4mb4z\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:27:18.006185 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:18.006150 4455 generic.go:356] "Generic (PLEG): container finished" podID="289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5" containerID="2bba86bbf62fd5652b4be287b7d42e5f2526b96c3842acfbb000a5e44c54195d" exitCode=0 Oct 30 18:27:18.006695 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:18.006211 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-6f78c8fc5f-p5dft" Oct 30 18:27:18.006695 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:18.006239 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-6f78c8fc5f-p5dft" event={"ID":"289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5","Type":"ContainerDied","Data":"2bba86bbf62fd5652b4be287b7d42e5f2526b96c3842acfbb000a5e44c54195d"} Oct 30 18:27:18.006695 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:18.006292 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-6f78c8fc5f-p5dft" event={"ID":"289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5","Type":"ContainerDied","Data":"71427838ba68e3870cf65abdd6394b55bb122fc3747b66ea5a1868af08f5019d"} Oct 30 18:27:18.006695 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:18.006369 4455 scope.go:122] "RemoveContainer" containerID="2bba86bbf62fd5652b4be287b7d42e5f2526b96c3842acfbb000a5e44c54195d" Oct 30 18:27:18.009349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:18.008907 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-66f64c68db-q5x68" Oct 30 18:27:18.009349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:18.009062 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-602e18-66f64c68db-q5x68" event={"ID":"a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9","Type":"ContainerDied","Data":"9468a66262389b855ea685914d78de7fe06543f30842f9db5e906e87d3581cb9"} Oct 30 18:27:18.021101 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:18.021077 4455 scope.go:122] "RemoveContainer" containerID="2bba86bbf62fd5652b4be287b7d42e5f2526b96c3842acfbb000a5e44c54195d" Oct 30 18:27:18.021450 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:27:18.021423 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bba86bbf62fd5652b4be287b7d42e5f2526b96c3842acfbb000a5e44c54195d\": container with ID starting with 2bba86bbf62fd5652b4be287b7d42e5f2526b96c3842acfbb000a5e44c54195d not found: ID does not exist" containerID="2bba86bbf62fd5652b4be287b7d42e5f2526b96c3842acfbb000a5e44c54195d" Oct 30 18:27:18.021569 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:18.021461 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bba86bbf62fd5652b4be287b7d42e5f2526b96c3842acfbb000a5e44c54195d"} err="failed to get container status \"2bba86bbf62fd5652b4be287b7d42e5f2526b96c3842acfbb000a5e44c54195d\": rpc error: code = NotFound desc = could not find container \"2bba86bbf62fd5652b4be287b7d42e5f2526b96c3842acfbb000a5e44c54195d\": container with ID starting with 2bba86bbf62fd5652b4be287b7d42e5f2526b96c3842acfbb000a5e44c54195d not found: ID does not exist" Oct 30 18:27:18.021569 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:18.021496 4455 scope.go:122] "RemoveContainer" containerID="8d229e713607a1b4eb29b5c08e4befc2085651183d4d68530966b1e677c48242" Oct 30 18:27:18.078841 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:18.078806 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-602e18-66f64c68db-q5x68"] Oct 30 18:27:18.096083 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:18.096010 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-602e18-66f64c68db-q5x68"] Oct 30 18:27:18.122603 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:18.122569 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-6f78c8fc5f-p5dft"] Oct 30 18:27:18.129305 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:18.129265 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-602e18-6f78c8fc5f-p5dft"] Oct 30 18:27:19.637405 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:19.637376 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5" path="/var/lib/kubelet/pods/289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5/volumes" Oct 30 18:27:19.637845 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:19.637831 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9" path="/var/lib/kubelet/pods/a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9/volumes" Oct 30 18:27:26.966943 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:26.966904 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g"] Oct 30 18:27:26.968845 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:26.968810 4455 cpu_manager.go:414] "containerMap: removing container" podUID="289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5" containerName="payload-pre-processing" Oct 30 18:27:26.968845 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:26.968844 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5" containerName="payload-pre-processing" Oct 30 18:27:26.969077 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:26.968874 4455 cpu_manager.go:414] "containerMap: removing container" podUID="bfbae34d-b472-4dd7-b862-dbb457726886" containerName="maas-api" Oct 30 18:27:26.969077 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:26.968883 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfbae34d-b472-4dd7-b862-dbb457726886" containerName="maas-api" Oct 30 18:27:26.969077 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:26.968896 4455 cpu_manager.go:414] "containerMap: removing container" podUID="46eee9bc-87c0-4421-92e7-547a494dbf3c" containerName="payload-pre-processing" Oct 30 18:27:26.969077 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:26.968904 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46eee9bc-87c0-4421-92e7-547a494dbf3c" containerName="payload-pre-processing" Oct 30 18:27:26.969077 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:26.968917 4455 cpu_manager.go:414] "containerMap: removing container" podUID="cb055040-07c9-46eb-8649-dd1db076ee59" containerName="istio-proxy" Oct 30 18:27:26.969077 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:26.968925 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb055040-07c9-46eb-8649-dd1db076ee59" containerName="istio-proxy" Oct 30 18:27:26.969077 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:26.968950 4455 cpu_manager.go:414] "containerMap: removing container" podUID="45acf61c-885f-4dc5-9ca3-d8293675107a" containerName="istio-proxy" Oct 30 18:27:26.969077 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:26.968957 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45acf61c-885f-4dc5-9ca3-d8293675107a" containerName="istio-proxy" Oct 30 18:27:26.969077 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:26.968974 4455 cpu_manager.go:414] "containerMap: removing container" podUID="6a166721-a635-4df9-adfe-7887eb5e7b39" containerName="payload-processing" Oct 30 18:27:26.969077 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:26.968980 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a166721-a635-4df9-adfe-7887eb5e7b39" containerName="payload-processing" Oct 30 18:27:26.969077 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:26.969002 4455 cpu_manager.go:414] "containerMap: removing container" podUID="da2b820f-4f95-41a7-a438-32c71a5e3679" containerName="maas-api" Oct 30 18:27:26.969077 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:26.969010 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da2b820f-4f95-41a7-a438-32c71a5e3679" containerName="maas-api" Oct 30 18:27:26.969077 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:26.969041 4455 cpu_manager.go:414] "containerMap: removing container" podUID="a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9" containerName="payload-processing" Oct 30 18:27:26.969077 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:26.969050 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9" containerName="payload-processing" Oct 30 18:27:26.969905 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:26.969230 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="289a6ea5-b4db-4a0b-b6fd-74f8e9f0ecb5" containerName="payload-pre-processing" Oct 30 18:27:26.969905 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:26.969253 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfbae34d-b472-4dd7-b862-dbb457726886" containerName="maas-api" Oct 30 18:27:26.969905 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:26.969264 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="46eee9bc-87c0-4421-92e7-547a494dbf3c" containerName="payload-pre-processing" Oct 30 18:27:26.969905 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:26.969272 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb055040-07c9-46eb-8649-dd1db076ee59" containerName="istio-proxy" Oct 30 18:27:26.969905 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:26.969285 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="45acf61c-885f-4dc5-9ca3-d8293675107a" containerName="istio-proxy" Oct 30 18:27:26.969905 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:26.969297 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a166721-a635-4df9-adfe-7887eb5e7b39" containerName="payload-processing" Oct 30 18:27:26.969905 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:26.969307 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="da2b820f-4f95-41a7-a438-32c71a5e3679" containerName="maas-api" Oct 30 18:27:26.969905 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:26.969341 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8c21d3e-9be0-48c6-95ba-1f1d8d186dd9" containerName="payload-processing" Oct 30 18:27:26.976646 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:26.976616 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" Oct 30 18:27:26.990209 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:26.990179 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g"] Oct 30 18:27:26.997431 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:26.997402 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-77db82\"/\"ipp-route-77db82-kserve-self-signed-certs\"" Oct 30 18:27:26.997593 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:26.997511 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-77db82\"/\"default-dockercfg-gzm5l\"" Oct 30 18:27:27.008970 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:27.008943 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-77db82\"/\"kube-root-ca.crt\"" Oct 30 18:27:27.009150 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:27.009009 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-77db82\"/\"openshift-service-ca.crt\"" Oct 30 18:27:27.050662 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:27.050626 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pddd\" (UniqueName: \"kubernetes.io/projected/25474977-dd58-4ead-9910-f20df0701ae5-kube-api-access-6pddd\") pod \"ipp-route-77db82-kserve-6464b5557-zxz9g\" (UID: \"25474977-dd58-4ead-9910-f20df0701ae5\") " pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" Oct 30 18:27:27.050850 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:27.050696 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25474977-dd58-4ead-9910-f20df0701ae5-home\") pod \"ipp-route-77db82-kserve-6464b5557-zxz9g\" (UID: \"25474977-dd58-4ead-9910-f20df0701ae5\") " pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" Oct 30 18:27:27.050850 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:27.050828 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25474977-dd58-4ead-9910-f20df0701ae5-tmp-dir\") pod \"ipp-route-77db82-kserve-6464b5557-zxz9g\" (UID: \"25474977-dd58-4ead-9910-f20df0701ae5\") " pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" Oct 30 18:27:27.051039 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:27.051019 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25474977-dd58-4ead-9910-f20df0701ae5-tls-certs\") pod \"ipp-route-77db82-kserve-6464b5557-zxz9g\" (UID: \"25474977-dd58-4ead-9910-f20df0701ae5\") " pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" Oct 30 18:27:27.051098 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:27.051059 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25474977-dd58-4ead-9910-f20df0701ae5-dshm\") pod \"ipp-route-77db82-kserve-6464b5557-zxz9g\" (UID: \"25474977-dd58-4ead-9910-f20df0701ae5\") " pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" Oct 30 18:27:27.051183 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:27.051163 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25474977-dd58-4ead-9910-f20df0701ae5-model-cache\") pod \"ipp-route-77db82-kserve-6464b5557-zxz9g\" (UID: \"25474977-dd58-4ead-9910-f20df0701ae5\") " pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" Oct 30 18:27:27.152739 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:27.152699 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25474977-dd58-4ead-9910-f20df0701ae5-tls-certs\") pod \"ipp-route-77db82-kserve-6464b5557-zxz9g\" (UID: \"25474977-dd58-4ead-9910-f20df0701ae5\") " pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" Oct 30 18:27:27.152739 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:27.152748 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25474977-dd58-4ead-9910-f20df0701ae5-dshm\") pod \"ipp-route-77db82-kserve-6464b5557-zxz9g\" (UID: \"25474977-dd58-4ead-9910-f20df0701ae5\") " pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" Oct 30 18:27:27.153018 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:27.152787 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25474977-dd58-4ead-9910-f20df0701ae5-model-cache\") pod \"ipp-route-77db82-kserve-6464b5557-zxz9g\" (UID: \"25474977-dd58-4ead-9910-f20df0701ae5\") " pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" Oct 30 18:27:27.153018 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:27.152868 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pddd\" (UniqueName: \"kubernetes.io/projected/25474977-dd58-4ead-9910-f20df0701ae5-kube-api-access-6pddd\") pod \"ipp-route-77db82-kserve-6464b5557-zxz9g\" (UID: \"25474977-dd58-4ead-9910-f20df0701ae5\") " pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" Oct 30 18:27:27.153018 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:27.152939 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25474977-dd58-4ead-9910-f20df0701ae5-home\") pod \"ipp-route-77db82-kserve-6464b5557-zxz9g\" (UID: \"25474977-dd58-4ead-9910-f20df0701ae5\") " pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" Oct 30 18:27:27.153018 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:27.152999 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25474977-dd58-4ead-9910-f20df0701ae5-tmp-dir\") pod \"ipp-route-77db82-kserve-6464b5557-zxz9g\" (UID: \"25474977-dd58-4ead-9910-f20df0701ae5\") " pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" Oct 30 18:27:27.153418 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:27.153390 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25474977-dd58-4ead-9910-f20df0701ae5-home\") pod \"ipp-route-77db82-kserve-6464b5557-zxz9g\" (UID: \"25474977-dd58-4ead-9910-f20df0701ae5\") " pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" Oct 30 18:27:27.153418 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:27.153408 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25474977-dd58-4ead-9910-f20df0701ae5-tmp-dir\") pod \"ipp-route-77db82-kserve-6464b5557-zxz9g\" (UID: \"25474977-dd58-4ead-9910-f20df0701ae5\") " pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" Oct 30 18:27:27.153754 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:27.153631 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25474977-dd58-4ead-9910-f20df0701ae5-model-cache\") pod \"ipp-route-77db82-kserve-6464b5557-zxz9g\" (UID: \"25474977-dd58-4ead-9910-f20df0701ae5\") " pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" Oct 30 18:27:27.155407 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:27.155378 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25474977-dd58-4ead-9910-f20df0701ae5-dshm\") pod \"ipp-route-77db82-kserve-6464b5557-zxz9g\" (UID: \"25474977-dd58-4ead-9910-f20df0701ae5\") " pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" Oct 30 18:27:27.155671 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:27.155649 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25474977-dd58-4ead-9910-f20df0701ae5-tls-certs\") pod \"ipp-route-77db82-kserve-6464b5557-zxz9g\" (UID: \"25474977-dd58-4ead-9910-f20df0701ae5\") " pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" Oct 30 18:27:27.247844 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:27.244740 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pddd\" (UniqueName: \"kubernetes.io/projected/25474977-dd58-4ead-9910-f20df0701ae5-kube-api-access-6pddd\") pod \"ipp-route-77db82-kserve-6464b5557-zxz9g\" (UID: \"25474977-dd58-4ead-9910-f20df0701ae5\") " pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" Oct 30 18:27:27.300347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:27.297673 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" Oct 30 18:27:27.998375 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:27:27.998248 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25474977_dd58_4ead_9910_f20df0701ae5.slice/crio-4552ae36b32d6af9d02e94423db9062e622f7fe616901cfdc139a70d5993eeb9 WatchSource:0}: Error finding container 4552ae36b32d6af9d02e94423db9062e622f7fe616901cfdc139a70d5993eeb9: Status 404 returned error can't find the container with id 4552ae36b32d6af9d02e94423db9062e622f7fe616901cfdc139a70d5993eeb9 Oct 30 18:27:27.998751 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:27.998622 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g"] Oct 30 18:27:28.095072 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:28.095042 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" event={"ID":"25474977-dd58-4ead-9910-f20df0701ae5","Type":"ContainerStarted","Data":"263a90bf4b60dc8b9c7e3ffb89c53dcd87c6ab9f02e54acb332be2b62f63e530"} Oct 30 18:27:28.095072 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:28.095086 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" event={"ID":"25474977-dd58-4ead-9910-f20df0701ae5","Type":"ContainerStarted","Data":"4552ae36b32d6af9d02e94423db9062e622f7fe616901cfdc139a70d5993eeb9"} Oct 30 18:27:28.146524 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:28.146466 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" podStartSLOduration=2.14645048 podStartE2EDuration="2.14645048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:27:28.143705314 +0000 UTC m=+2100.631342349" watchObservedRunningTime="2026-10-30 18:27:28.14645048 +0000 UTC m=+2100.634087513" Oct 30 18:27:28.303804 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:28.303718 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" Oct 30 18:27:30.391551 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:27:30.391512 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:27:37.298548 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:37.298505 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" Oct 30 18:27:37.316591 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:37.316561 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" Oct 30 18:27:38.228787 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:27:38.228760 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" Oct 30 18:28:00.385581 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:28:00.385542 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:28:26.069347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:26.067053 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:26.263965 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:26.263911 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" podUID="25474977-dd58-4ead-9910-f20df0701ae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.184:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:27.076661 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:27.073001 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:28:27.076661 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:27.076634 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:28:27.289348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:27.278386 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" podUID="25474977-dd58-4ead-9910-f20df0701ae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.184:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:27.408403 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:27.384598 4455 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:27.492641 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:27.416089 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" Oct 30 18:28:28.139623 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:28.139579 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:28.312732 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:28.312415 4455 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" podUID="25474977-dd58-4ead-9910-f20df0701ae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.184:8000/health\": context deadline exceeded" Oct 30 18:28:28.418408 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:28.409684 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" podUID="25474977-dd58-4ead-9910-f20df0701ae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.184:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:28.617088 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:28.617011 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:28:28.617436 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:28.617400 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" podUID="c7c21be6-b9eb-401c-aa3c-af5b62960cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:29.156350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:29.155924 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:28:29.253521 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:29.253481 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:28:29.253838 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:29.253797 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" podUID="4e7b0c73-0041-4549-b47e-76a7aaed2b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:29.410271 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:29.410148 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" podUID="25474977-dd58-4ead-9910-f20df0701ae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.184:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:29.981953 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:29.981919 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:28:30.267402 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:30.263709 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" Oct 30 18:28:30.651407 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:28:30.630491 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:28:31.291345 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:31.290825 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:28:31.291345 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:31.290895 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" podUID="1d3471dc-aaf3-4060-a64d-2413e522e65a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:54.926738 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:54.926693 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ac632e-7d5967f8dc-blb7b"] Oct 30 18:28:54.932556 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:54.932531 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ac632e-7d5967f8dc-blb7b" Oct 30 18:28:54.952366 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:54.952312 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ac632e-7d5967f8dc-blb7b"] Oct 30 18:28:54.961665 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:54.961634 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-ac632e\"" Oct 30 18:28:54.961665 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:54.961652 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-ac632e-dockercfg-vzv89\"" Oct 30 18:28:55.047256 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:55.047203 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/670e8cf0-8eb6-4480-821c-3ead1b573195-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-ac632e-7d5967f8dc-blb7b\" (UID: \"670e8cf0-8eb6-4480-821c-3ead1b573195\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ac632e-7d5967f8dc-blb7b" Oct 30 18:28:55.047750 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:55.047714 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vzf7\" (UniqueName: \"kubernetes.io/projected/670e8cf0-8eb6-4480-821c-3ead1b573195-kube-api-access-9vzf7\") pod \"payload-pre-processing-e2e-shared-a-w0-ac632e-7d5967f8dc-blb7b\" (UID: \"670e8cf0-8eb6-4480-821c-3ead1b573195\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ac632e-7d5967f8dc-blb7b" Oct 30 18:28:55.173797 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:55.173754 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/670e8cf0-8eb6-4480-821c-3ead1b573195-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-ac632e-7d5967f8dc-blb7b\" (UID: \"670e8cf0-8eb6-4480-821c-3ead1b573195\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ac632e-7d5967f8dc-blb7b" Oct 30 18:28:55.175536 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:55.175509 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vzf7\" (UniqueName: \"kubernetes.io/projected/670e8cf0-8eb6-4480-821c-3ead1b573195-kube-api-access-9vzf7\") pod \"payload-pre-processing-e2e-shared-a-w0-ac632e-7d5967f8dc-blb7b\" (UID: \"670e8cf0-8eb6-4480-821c-3ead1b573195\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ac632e-7d5967f8dc-blb7b" Oct 30 18:28:55.186470 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:55.175259 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/670e8cf0-8eb6-4480-821c-3ead1b573195-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-ac632e-7d5967f8dc-blb7b\" (UID: \"670e8cf0-8eb6-4480-821c-3ead1b573195\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ac632e-7d5967f8dc-blb7b" Oct 30 18:28:55.248073 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:55.247542 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ac632e-75c75cf4cb-nvwzm"] Oct 30 18:28:55.273173 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:55.273002 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vzf7\" (UniqueName: \"kubernetes.io/projected/670e8cf0-8eb6-4480-821c-3ead1b573195-kube-api-access-9vzf7\") pod \"payload-pre-processing-e2e-shared-a-w0-ac632e-7d5967f8dc-blb7b\" (UID: \"670e8cf0-8eb6-4480-821c-3ead1b573195\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ac632e-7d5967f8dc-blb7b" Oct 30 18:28:55.287447 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:55.286703 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ac632e-75c75cf4cb-nvwzm"] Oct 30 18:28:55.287447 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:55.286874 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ac632e-75c75cf4cb-nvwzm" Oct 30 18:28:55.389899 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:55.389854 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7be0ce35-cc15-480d-aaed-51c7a709b124-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-ac632e-75c75cf4cb-nvwzm\" (UID: \"7be0ce35-cc15-480d-aaed-51c7a709b124\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ac632e-75c75cf4cb-nvwzm" Oct 30 18:28:55.390084 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:55.390060 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtsxr\" (UniqueName: \"kubernetes.io/projected/7be0ce35-cc15-480d-aaed-51c7a709b124-kube-api-access-qtsxr\") pod \"payload-processing-e2e-shared-a-w0-ac632e-75c75cf4cb-nvwzm\" (UID: \"7be0ce35-cc15-480d-aaed-51c7a709b124\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ac632e-75c75cf4cb-nvwzm" Oct 30 18:28:55.511603 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:55.508240 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtsxr\" (UniqueName: \"kubernetes.io/projected/7be0ce35-cc15-480d-aaed-51c7a709b124-kube-api-access-qtsxr\") pod \"payload-processing-e2e-shared-a-w0-ac632e-75c75cf4cb-nvwzm\" (UID: \"7be0ce35-cc15-480d-aaed-51c7a709b124\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ac632e-75c75cf4cb-nvwzm" Oct 30 18:28:55.511603 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:55.508567 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7be0ce35-cc15-480d-aaed-51c7a709b124-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-ac632e-75c75cf4cb-nvwzm\" (UID: \"7be0ce35-cc15-480d-aaed-51c7a709b124\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ac632e-75c75cf4cb-nvwzm" Oct 30 18:28:55.534351 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:55.532613 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7be0ce35-cc15-480d-aaed-51c7a709b124-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-ac632e-75c75cf4cb-nvwzm\" (UID: \"7be0ce35-cc15-480d-aaed-51c7a709b124\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ac632e-75c75cf4cb-nvwzm" Oct 30 18:28:55.558352 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:55.557698 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ac632e-7d5967f8dc-blb7b" Oct 30 18:28:55.558817 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:55.558782 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtsxr\" (UniqueName: \"kubernetes.io/projected/7be0ce35-cc15-480d-aaed-51c7a709b124-kube-api-access-qtsxr\") pod \"payload-processing-e2e-shared-a-w0-ac632e-75c75cf4cb-nvwzm\" (UID: \"7be0ce35-cc15-480d-aaed-51c7a709b124\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ac632e-75c75cf4cb-nvwzm" Oct 30 18:28:55.621704 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:55.620591 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ac632e-75c75cf4cb-nvwzm" Oct 30 18:28:55.907253 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:55.907221 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ac632e-7d5967f8dc-blb7b"] Oct 30 18:28:55.907780 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:28:55.907757 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod670e8cf0_8eb6_4480_821c_3ead1b573195.slice/crio-48ad4645ac95036c6f3051018f9b3401628db6584b36cfdcc4d1ed4aed0be985 WatchSource:0}: Error finding container 48ad4645ac95036c6f3051018f9b3401628db6584b36cfdcc4d1ed4aed0be985: Status 404 returned error can't find the container with id 48ad4645ac95036c6f3051018f9b3401628db6584b36cfdcc4d1ed4aed0be985 Oct 30 18:28:55.971556 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:28:55.971526 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7be0ce35_cc15_480d_aaed_51c7a709b124.slice/crio-ba8f66b28ac92b3cf35596c4c9704abec45b23ca87f1752ac5d821f89f3f42d6 WatchSource:0}: Error finding container ba8f66b28ac92b3cf35596c4c9704abec45b23ca87f1752ac5d821f89f3f42d6: Status 404 returned error can't find the container with id ba8f66b28ac92b3cf35596c4c9704abec45b23ca87f1752ac5d821f89f3f42d6 Oct 30 18:28:55.972703 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:55.972676 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ac632e-75c75cf4cb-nvwzm"] Oct 30 18:28:56.317921 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:56.317540 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ac632e-75c75cf4cb-nvwzm" event={"ID":"7be0ce35-cc15-480d-aaed-51c7a709b124","Type":"ContainerStarted","Data":"b2468bdaba22afca15fa75643bdda8aa54241d4e4d0c1fc0d588229723462d57"} Oct 30 18:28:56.317921 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:56.317620 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ac632e-75c75cf4cb-nvwzm" Oct 30 18:28:56.317921 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:56.317648 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ac632e-75c75cf4cb-nvwzm" event={"ID":"7be0ce35-cc15-480d-aaed-51c7a709b124","Type":"ContainerStarted","Data":"ba8f66b28ac92b3cf35596c4c9704abec45b23ca87f1752ac5d821f89f3f42d6"} Oct 30 18:28:56.336222 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:56.319780 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ac632e-7d5967f8dc-blb7b" event={"ID":"670e8cf0-8eb6-4480-821c-3ead1b573195","Type":"ContainerStarted","Data":"df350166a98e75c20eb18ef0b7c73d36e48b21c1b80a2b19da57751f94374fcf"} Oct 30 18:28:56.336222 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:56.319832 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ac632e-7d5967f8dc-blb7b" event={"ID":"670e8cf0-8eb6-4480-821c-3ead1b573195","Type":"ContainerStarted","Data":"48ad4645ac95036c6f3051018f9b3401628db6584b36cfdcc4d1ed4aed0be985"} Oct 30 18:28:56.336222 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:56.319879 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ac632e-7d5967f8dc-blb7b" Oct 30 18:28:56.371041 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:56.370981 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ac632e-75c75cf4cb-nvwzm" podStartSLOduration=1.370963278 podStartE2EDuration="1.370963278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:28:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:28:56.369236748 +0000 UTC m=+2188.856873784" watchObservedRunningTime="2026-10-30 18:28:56.370963278 +0000 UTC m=+2188.858600312" Oct 30 18:28:56.416289 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:28:56.416226 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ac632e-7d5967f8dc-blb7b" podStartSLOduration=2.416211367 podStartE2EDuration="2.416211367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:28:56.412227279 +0000 UTC m=+2188.899864313" watchObservedRunningTime="2026-10-30 18:28:56.416211367 +0000 UTC m=+2188.903848453" Oct 30 18:29:00.385657 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:29:00.385614 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:29:07.330144 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:07.330113 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ac632e-7d5967f8dc-blb7b" Oct 30 18:29:07.330766 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:07.330180 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ac632e-75c75cf4cb-nvwzm" Oct 30 18:29:07.394155 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:07.394115 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg"] Oct 30 18:29:07.427854 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:07.427815 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg" Oct 30 18:29:07.428045 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:07.427947 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg"] Oct 30 18:29:07.444645 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:07.444612 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-ac632e-serving-cert\"" Oct 30 18:29:07.557649 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:07.557613 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37f249f2-67f0-4ba7-bb80-05d7c6e698e9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg\" (UID: \"37f249f2-67f0-4ba7-bb80-05d7c6e698e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg" Oct 30 18:29:07.557829 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:07.557684 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j47wv\" (UniqueName: \"kubernetes.io/projected/37f249f2-67f0-4ba7-bb80-05d7c6e698e9-kube-api-access-j47wv\") pod \"maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg\" (UID: \"37f249f2-67f0-4ba7-bb80-05d7c6e698e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg" Oct 30 18:29:07.557874 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:07.557819 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37f249f2-67f0-4ba7-bb80-05d7c6e698e9-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg\" (UID: \"37f249f2-67f0-4ba7-bb80-05d7c6e698e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg" Oct 30 18:29:07.557998 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:07.557979 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37f249f2-67f0-4ba7-bb80-05d7c6e698e9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg\" (UID: \"37f249f2-67f0-4ba7-bb80-05d7c6e698e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg" Oct 30 18:29:07.666413 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:07.659985 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j47wv\" (UniqueName: \"kubernetes.io/projected/37f249f2-67f0-4ba7-bb80-05d7c6e698e9-kube-api-access-j47wv\") pod \"maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg\" (UID: \"37f249f2-67f0-4ba7-bb80-05d7c6e698e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg" Oct 30 18:29:07.666413 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:07.660121 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37f249f2-67f0-4ba7-bb80-05d7c6e698e9-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg\" (UID: \"37f249f2-67f0-4ba7-bb80-05d7c6e698e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg" Oct 30 18:29:07.666413 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:07.660277 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37f249f2-67f0-4ba7-bb80-05d7c6e698e9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg\" (UID: \"37f249f2-67f0-4ba7-bb80-05d7c6e698e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg" Oct 30 18:29:07.666413 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:07.660504 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37f249f2-67f0-4ba7-bb80-05d7c6e698e9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg\" (UID: \"37f249f2-67f0-4ba7-bb80-05d7c6e698e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg" Oct 30 18:29:07.666413 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:07.662371 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37f249f2-67f0-4ba7-bb80-05d7c6e698e9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg\" (UID: \"37f249f2-67f0-4ba7-bb80-05d7c6e698e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg" Oct 30 18:29:07.695742 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:07.691536 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37f249f2-67f0-4ba7-bb80-05d7c6e698e9-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg\" (UID: \"37f249f2-67f0-4ba7-bb80-05d7c6e698e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg" Oct 30 18:29:07.698830 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:07.696977 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37f249f2-67f0-4ba7-bb80-05d7c6e698e9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg\" (UID: \"37f249f2-67f0-4ba7-bb80-05d7c6e698e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg" Oct 30 18:29:07.705221 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:07.705185 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j47wv\" (UniqueName: \"kubernetes.io/projected/37f249f2-67f0-4ba7-bb80-05d7c6e698e9-kube-api-access-j47wv\") pod \"maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg\" (UID: \"37f249f2-67f0-4ba7-bb80-05d7c6e698e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg" Oct 30 18:29:07.748671 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:07.748641 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg" Oct 30 18:29:08.066613 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:08.066577 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg"] Oct 30 18:29:08.067180 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:29:08.067152 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f249f2_67f0_4ba7_bb80_05d7c6e698e9.slice/crio-882fd8299500d3835b4c3b376723bbf3e02642fe2099089f835cfbd21564de6a WatchSource:0}: Error finding container 882fd8299500d3835b4c3b376723bbf3e02642fe2099089f835cfbd21564de6a: Status 404 returned error can't find the container with id 882fd8299500d3835b4c3b376723bbf3e02642fe2099089f835cfbd21564de6a Oct 30 18:29:08.072177 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:08.072013 4455 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:29:08.447154 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:08.447124 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg" event={"ID":"37f249f2-67f0-4ba7-bb80-05d7c6e698e9","Type":"ContainerStarted","Data":"882fd8299500d3835b4c3b376723bbf3e02642fe2099089f835cfbd21564de6a"} Oct 30 18:29:09.455780 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:09.455725 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg" event={"ID":"37f249f2-67f0-4ba7-bb80-05d7c6e698e9","Type":"ContainerStarted","Data":"18aec4832db0ae88a6570d183a511bfebb79d187f793a6c8f78d81c09b999e5d"} Oct 30 18:29:09.456198 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:09.455913 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg" Oct 30 18:29:09.562503 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:09.559628 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg" podStartSLOduration=2.205993205 podStartE2EDuration="2.559605765s" totalImagesPullingTime="353.61256ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:09.514853355 +0000 UTC m=+2202.002490405" watchObservedRunningTime="2026-10-30 18:29:09.559605765 +0000 UTC m=+2202.047242802" Oct 30 18:29:11.148302 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:11.148262 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a2a10a-5f7758695-k2fz7"] Oct 30 18:29:11.153479 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:11.153454 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a2a10a-5f7758695-k2fz7" Oct 30 18:29:11.176460 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:11.176425 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a2a10a-5f7758695-k2fz7"] Oct 30 18:29:11.183817 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:11.183789 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-a2a10a-dockercfg-2srk9\"" Oct 30 18:29:11.183987 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:11.183856 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-a2a10a\"" Oct 30 18:29:11.356346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:11.321194 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e30ec8f2-cffa-4713-823c-ce00bbcae4ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-a2a10a-5f7758695-k2fz7\" (UID: \"e30ec8f2-cffa-4713-823c-ce00bbcae4ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a2a10a-5f7758695-k2fz7" Oct 30 18:29:11.356683 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:11.356656 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqmp4\" (UniqueName: \"kubernetes.io/projected/e30ec8f2-cffa-4713-823c-ce00bbcae4ad-kube-api-access-cqmp4\") pod \"payload-pre-processing-e2e-shared-a-w2-a2a10a-5f7758695-k2fz7\" (UID: \"e30ec8f2-cffa-4713-823c-ce00bbcae4ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a2a10a-5f7758695-k2fz7" Oct 30 18:29:11.440512 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:11.432261 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a2a10a-c7c86844-xd2qw"] Oct 30 18:29:11.474347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:11.463865 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e30ec8f2-cffa-4713-823c-ce00bbcae4ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-a2a10a-5f7758695-k2fz7\" (UID: \"e30ec8f2-cffa-4713-823c-ce00bbcae4ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a2a10a-5f7758695-k2fz7" Oct 30 18:29:11.483523 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:11.475475 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqmp4\" (UniqueName: \"kubernetes.io/projected/e30ec8f2-cffa-4713-823c-ce00bbcae4ad-kube-api-access-cqmp4\") pod \"payload-pre-processing-e2e-shared-a-w2-a2a10a-5f7758695-k2fz7\" (UID: \"e30ec8f2-cffa-4713-823c-ce00bbcae4ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a2a10a-5f7758695-k2fz7" Oct 30 18:29:11.484016 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:11.483964 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e30ec8f2-cffa-4713-823c-ce00bbcae4ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-a2a10a-5f7758695-k2fz7\" (UID: \"e30ec8f2-cffa-4713-823c-ce00bbcae4ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a2a10a-5f7758695-k2fz7" Oct 30 18:29:11.486784 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:11.486756 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a2a10a-c7c86844-xd2qw" Oct 30 18:29:11.531400 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:11.519879 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a2a10a-c7c86844-xd2qw"] Oct 30 18:29:11.531400 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:11.530018 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqmp4\" (UniqueName: \"kubernetes.io/projected/e30ec8f2-cffa-4713-823c-ce00bbcae4ad-kube-api-access-cqmp4\") pod \"payload-pre-processing-e2e-shared-a-w2-a2a10a-5f7758695-k2fz7\" (UID: \"e30ec8f2-cffa-4713-823c-ce00bbcae4ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a2a10a-5f7758695-k2fz7" Oct 30 18:29:11.577273 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:11.577223 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbx6g\" (UniqueName: \"kubernetes.io/projected/774b05b3-b9f0-412d-a530-d962c7344079-kube-api-access-cbx6g\") pod \"payload-processing-e2e-shared-a-w2-a2a10a-c7c86844-xd2qw\" (UID: \"774b05b3-b9f0-412d-a530-d962c7344079\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a2a10a-c7c86844-xd2qw" Oct 30 18:29:11.577514 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:11.577337 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/774b05b3-b9f0-412d-a530-d962c7344079-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-a2a10a-c7c86844-xd2qw\" (UID: \"774b05b3-b9f0-412d-a530-d962c7344079\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a2a10a-c7c86844-xd2qw" Oct 30 18:29:11.679312 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:11.679274 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbx6g\" (UniqueName: \"kubernetes.io/projected/774b05b3-b9f0-412d-a530-d962c7344079-kube-api-access-cbx6g\") pod \"payload-processing-e2e-shared-a-w2-a2a10a-c7c86844-xd2qw\" (UID: \"774b05b3-b9f0-412d-a530-d962c7344079\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a2a10a-c7c86844-xd2qw" Oct 30 18:29:11.679526 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:11.679385 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/774b05b3-b9f0-412d-a530-d962c7344079-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-a2a10a-c7c86844-xd2qw\" (UID: \"774b05b3-b9f0-412d-a530-d962c7344079\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a2a10a-c7c86844-xd2qw" Oct 30 18:29:11.680373 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:11.680350 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/774b05b3-b9f0-412d-a530-d962c7344079-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-a2a10a-c7c86844-xd2qw\" (UID: \"774b05b3-b9f0-412d-a530-d962c7344079\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a2a10a-c7c86844-xd2qw" Oct 30 18:29:11.704420 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:11.704314 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbx6g\" (UniqueName: \"kubernetes.io/projected/774b05b3-b9f0-412d-a530-d962c7344079-kube-api-access-cbx6g\") pod \"payload-processing-e2e-shared-a-w2-a2a10a-c7c86844-xd2qw\" (UID: \"774b05b3-b9f0-412d-a530-d962c7344079\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a2a10a-c7c86844-xd2qw" Oct 30 18:29:11.770598 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:11.770559 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a2a10a-5f7758695-k2fz7" Oct 30 18:29:11.862267 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:11.857513 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a2a10a-c7c86844-xd2qw" Oct 30 18:29:12.133399 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:29:12.132358 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode30ec8f2_cffa_4713_823c_ce00bbcae4ad.slice/crio-37aac5d231db4206f559ad3dd2bbb23d311a701237caee440be6b24b9134942e WatchSource:0}: Error finding container 37aac5d231db4206f559ad3dd2bbb23d311a701237caee440be6b24b9134942e: Status 404 returned error can't find the container with id 37aac5d231db4206f559ad3dd2bbb23d311a701237caee440be6b24b9134942e Oct 30 18:29:12.135017 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:12.134991 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a2a10a-5f7758695-k2fz7"] Oct 30 18:29:12.202122 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:12.202089 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a2a10a-c7c86844-xd2qw"] Oct 30 18:29:12.205974 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:29:12.205948 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod774b05b3_b9f0_412d_a530_d962c7344079.slice/crio-495d0c5a8b38c5c975e2cdbc290e9e740e9bb1885e4039278dbdff1928843e74 WatchSource:0}: Error finding container 495d0c5a8b38c5c975e2cdbc290e9e740e9bb1885e4039278dbdff1928843e74: Status 404 returned error can't find the container with id 495d0c5a8b38c5c975e2cdbc290e9e740e9bb1885e4039278dbdff1928843e74 Oct 30 18:29:12.544988 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:12.544957 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a2a10a-c7c86844-xd2qw" event={"ID":"774b05b3-b9f0-412d-a530-d962c7344079","Type":"ContainerStarted","Data":"a33608f42c6220ace09efcde794bbfa9c19408b0cb66ead69225b9de64563154"} Oct 30 18:29:12.545176 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:12.545012 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a2a10a-c7c86844-xd2qw" event={"ID":"774b05b3-b9f0-412d-a530-d962c7344079","Type":"ContainerStarted","Data":"495d0c5a8b38c5c975e2cdbc290e9e740e9bb1885e4039278dbdff1928843e74"} Oct 30 18:29:12.545176 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:12.545038 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a2a10a-c7c86844-xd2qw" Oct 30 18:29:12.546765 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:12.546738 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a2a10a-5f7758695-k2fz7" event={"ID":"e30ec8f2-cffa-4713-823c-ce00bbcae4ad","Type":"ContainerStarted","Data":"b77b744dd9e53475d4bcb10cb9c5be2ada9ffef5018275d48492e82b3cd1f3e9"} Oct 30 18:29:12.546879 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:12.546779 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a2a10a-5f7758695-k2fz7" event={"ID":"e30ec8f2-cffa-4713-823c-ce00bbcae4ad","Type":"ContainerStarted","Data":"37aac5d231db4206f559ad3dd2bbb23d311a701237caee440be6b24b9134942e"} Oct 30 18:29:12.546879 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:12.546805 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a2a10a-5f7758695-k2fz7" Oct 30 18:29:12.591937 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:12.591873 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a2a10a-c7c86844-xd2qw" podStartSLOduration=1.5918532779999999 podStartE2EDuration="1.591853278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:12.588244234 +0000 UTC m=+2205.075881269" watchObservedRunningTime="2026-10-30 18:29:12.591853278 +0000 UTC m=+2205.079490313" Oct 30 18:29:12.634380 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:12.634302 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a2a10a-5f7758695-k2fz7" podStartSLOduration=1.634288674 podStartE2EDuration="1.634288674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:12.632225241 +0000 UTC m=+2205.119862272" watchObservedRunningTime="2026-10-30 18:29:12.634288674 +0000 UTC m=+2205.121925707" Oct 30 18:29:15.468834 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:15.468801 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg" Oct 30 18:29:23.561152 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:23.561104 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a2a10a-c7c86844-xd2qw" Oct 30 18:29:23.561975 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:23.561954 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a2a10a-5f7758695-k2fz7" Oct 30 18:29:24.374903 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:24.374869 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e77b1d-9fdccb855-cfxfl"] Oct 30 18:29:24.379809 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:24.379782 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e77b1d-9fdccb855-cfxfl" Oct 30 18:29:24.400685 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:24.400656 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a8b3a3-4d72-4077-875f-a0262982c7e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-e77b1d-9fdccb855-cfxfl\" (UID: \"67a8b3a3-4d72-4077-875f-a0262982c7e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e77b1d-9fdccb855-cfxfl" Oct 30 18:29:24.400935 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:24.400733 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sppl8\" (UniqueName: \"kubernetes.io/projected/67a8b3a3-4d72-4077-875f-a0262982c7e9-kube-api-access-sppl8\") pod \"payload-pre-processing-e2e-shared-a-w3-e77b1d-9fdccb855-cfxfl\" (UID: \"67a8b3a3-4d72-4077-875f-a0262982c7e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e77b1d-9fdccb855-cfxfl" Oct 30 18:29:24.401978 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:24.401940 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e77b1d-9fdccb855-cfxfl"] Oct 30 18:29:24.410439 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:24.410410 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-e77b1d-dockercfg-9rw2d\"" Oct 30 18:29:24.410439 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:24.410429 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-e77b1d\"" Oct 30 18:29:24.535358 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:24.506775 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a8b3a3-4d72-4077-875f-a0262982c7e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-e77b1d-9fdccb855-cfxfl\" (UID: \"67a8b3a3-4d72-4077-875f-a0262982c7e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e77b1d-9fdccb855-cfxfl" Oct 30 18:29:24.535792 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:24.535762 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sppl8\" (UniqueName: \"kubernetes.io/projected/67a8b3a3-4d72-4077-875f-a0262982c7e9-kube-api-access-sppl8\") pod \"payload-pre-processing-e2e-shared-a-w3-e77b1d-9fdccb855-cfxfl\" (UID: \"67a8b3a3-4d72-4077-875f-a0262982c7e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e77b1d-9fdccb855-cfxfl" Oct 30 18:29:24.536611 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:24.528031 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a8b3a3-4d72-4077-875f-a0262982c7e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-e77b1d-9fdccb855-cfxfl\" (UID: \"67a8b3a3-4d72-4077-875f-a0262982c7e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e77b1d-9fdccb855-cfxfl" Oct 30 18:29:24.602766 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:24.601167 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sppl8\" (UniqueName: \"kubernetes.io/projected/67a8b3a3-4d72-4077-875f-a0262982c7e9-kube-api-access-sppl8\") pod \"payload-pre-processing-e2e-shared-a-w3-e77b1d-9fdccb855-cfxfl\" (UID: \"67a8b3a3-4d72-4077-875f-a0262982c7e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e77b1d-9fdccb855-cfxfl" Oct 30 18:29:24.633175 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:24.629088 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e77b1d-55ddb87958-l9wgq"] Oct 30 18:29:24.635578 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:24.635543 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e77b1d-55ddb87958-l9wgq" Oct 30 18:29:24.644386 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:24.644348 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e77b1d-55ddb87958-l9wgq"] Oct 30 18:29:24.695570 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:24.695265 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e77b1d-9fdccb855-cfxfl" Oct 30 18:29:24.844835 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:24.844794 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vqtt\" (UniqueName: \"kubernetes.io/projected/c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff-kube-api-access-2vqtt\") pod \"payload-processing-e2e-shared-a-w3-e77b1d-55ddb87958-l9wgq\" (UID: \"c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e77b1d-55ddb87958-l9wgq" Oct 30 18:29:24.844966 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:24.844922 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-e77b1d-55ddb87958-l9wgq\" (UID: \"c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e77b1d-55ddb87958-l9wgq" Oct 30 18:29:24.946898 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:24.946796 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vqtt\" (UniqueName: \"kubernetes.io/projected/c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff-kube-api-access-2vqtt\") pod \"payload-processing-e2e-shared-a-w3-e77b1d-55ddb87958-l9wgq\" (UID: \"c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e77b1d-55ddb87958-l9wgq" Oct 30 18:29:24.947650 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:24.947619 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-e77b1d-55ddb87958-l9wgq\" (UID: \"c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e77b1d-55ddb87958-l9wgq" Oct 30 18:29:24.948584 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:24.948556 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-e77b1d-55ddb87958-l9wgq\" (UID: \"c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e77b1d-55ddb87958-l9wgq" Oct 30 18:29:24.973106 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:24.973061 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vqtt\" (UniqueName: \"kubernetes.io/projected/c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff-kube-api-access-2vqtt\") pod \"payload-processing-e2e-shared-a-w3-e77b1d-55ddb87958-l9wgq\" (UID: \"c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e77b1d-55ddb87958-l9wgq" Oct 30 18:29:25.015870 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:25.015828 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e77b1d-55ddb87958-l9wgq" Oct 30 18:29:25.106839 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:25.102731 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e77b1d-9fdccb855-cfxfl"] Oct 30 18:29:25.106839 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:29:25.106075 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a8b3a3_4d72_4077_875f_a0262982c7e9.slice/crio-74cd0b4ecacaf4f55f899279ba26bd1b3171b05243a1f679193e5c81d5f216bb WatchSource:0}: Error finding container 74cd0b4ecacaf4f55f899279ba26bd1b3171b05243a1f679193e5c81d5f216bb: Status 404 returned error can't find the container with id 74cd0b4ecacaf4f55f899279ba26bd1b3171b05243a1f679193e5c81d5f216bb Oct 30 18:29:25.339894 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:25.339864 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e77b1d-55ddb87958-l9wgq"] Oct 30 18:29:25.811380 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:25.811300 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e77b1d-9fdccb855-cfxfl" event={"ID":"67a8b3a3-4d72-4077-875f-a0262982c7e9","Type":"ContainerStarted","Data":"be21ba11cb5649a4342cf5d3d9e4e3776184392c0f31d3e9a1a52036c04a6133"} Oct 30 18:29:25.811828 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:25.811515 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e77b1d-9fdccb855-cfxfl" Oct 30 18:29:25.811828 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:25.811541 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e77b1d-9fdccb855-cfxfl" event={"ID":"67a8b3a3-4d72-4077-875f-a0262982c7e9","Type":"ContainerStarted","Data":"74cd0b4ecacaf4f55f899279ba26bd1b3171b05243a1f679193e5c81d5f216bb"} Oct 30 18:29:25.813239 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:25.813213 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e77b1d-55ddb87958-l9wgq" event={"ID":"c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff","Type":"ContainerStarted","Data":"96d33ad4cbde6ac358f6bc6d784f51c7ea8ad430d5bdbe28109bcc563cef6b34"} Oct 30 18:29:25.813239 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:25.813251 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e77b1d-55ddb87958-l9wgq" event={"ID":"c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff","Type":"ContainerStarted","Data":"d20c6dd8b3fa9d0748452cbf744bf6fca73fc423d0c69988c32d37e09db69145"} Oct 30 18:29:25.813449 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:25.813311 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e77b1d-55ddb87958-l9wgq" Oct 30 18:29:25.855702 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:25.855651 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e77b1d-9fdccb855-cfxfl" podStartSLOduration=1.8556126210000001 podStartE2EDuration="1.855612621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:25.855209409 +0000 UTC m=+2218.342846443" watchObservedRunningTime="2026-10-30 18:29:25.855612621 +0000 UTC m=+2218.343249654" Oct 30 18:29:25.904808 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:25.904743 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e77b1d-55ddb87958-l9wgq" podStartSLOduration=1.904725408 podStartE2EDuration="1.904725408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:25.902791226 +0000 UTC m=+2218.390428265" watchObservedRunningTime="2026-10-30 18:29:25.904725408 +0000 UTC m=+2218.392362444" Oct 30 18:29:26.462498 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:26.462460 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-50e9fc-56cdb5789f-h86nr"] Oct 30 18:29:26.468422 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:26.468395 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-50e9fc-56cdb5789f-h86nr" Oct 30 18:29:26.485722 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:26.484720 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-50e9fc\"" Oct 30 18:29:26.489308 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:26.489285 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-50e9fc-56cdb5789f-h86nr"] Oct 30 18:29:26.497754 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:26.497731 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-50e9fc-dockercfg-kx7pq\"" Oct 30 18:29:26.763410 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:26.751887 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13f59414-113d-4863-88b8-e43c25b17cff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-50e9fc-56cdb5789f-h86nr\" (UID: \"13f59414-113d-4863-88b8-e43c25b17cff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-50e9fc-56cdb5789f-h86nr" Oct 30 18:29:26.763410 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:26.752123 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srcm8\" (UniqueName: \"kubernetes.io/projected/13f59414-113d-4863-88b8-e43c25b17cff-kube-api-access-srcm8\") pod \"payload-pre-processing-e2e-shared-a-w4-50e9fc-56cdb5789f-h86nr\" (UID: \"13f59414-113d-4863-88b8-e43c25b17cff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-50e9fc-56cdb5789f-h86nr" Oct 30 18:29:26.839389 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:26.834253 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-50e9fc-647b8c7b4-78rvk"] Oct 30 18:29:26.851668 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:26.851635 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-50e9fc-647b8c7b4-78rvk"] Oct 30 18:29:26.852031 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:26.852008 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-50e9fc-647b8c7b4-78rvk" Oct 30 18:29:26.865373 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:26.865159 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz28f\" (UniqueName: \"kubernetes.io/projected/67ec911d-b41b-49f5-b1e3-4637bc7c48f3-kube-api-access-kz28f\") pod \"payload-processing-e2e-shared-a-w4-50e9fc-647b8c7b4-78rvk\" (UID: \"67ec911d-b41b-49f5-b1e3-4637bc7c48f3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-50e9fc-647b8c7b4-78rvk" Oct 30 18:29:26.867777 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:26.865687 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13f59414-113d-4863-88b8-e43c25b17cff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-50e9fc-56cdb5789f-h86nr\" (UID: \"13f59414-113d-4863-88b8-e43c25b17cff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-50e9fc-56cdb5789f-h86nr" Oct 30 18:29:26.867777 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:26.865819 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ec911d-b41b-49f5-b1e3-4637bc7c48f3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-50e9fc-647b8c7b4-78rvk\" (UID: \"67ec911d-b41b-49f5-b1e3-4637bc7c48f3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-50e9fc-647b8c7b4-78rvk" Oct 30 18:29:26.867777 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:26.865972 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srcm8\" (UniqueName: \"kubernetes.io/projected/13f59414-113d-4863-88b8-e43c25b17cff-kube-api-access-srcm8\") pod \"payload-pre-processing-e2e-shared-a-w4-50e9fc-56cdb5789f-h86nr\" (UID: \"13f59414-113d-4863-88b8-e43c25b17cff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-50e9fc-56cdb5789f-h86nr" Oct 30 18:29:26.867777 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:26.867711 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13f59414-113d-4863-88b8-e43c25b17cff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-50e9fc-56cdb5789f-h86nr\" (UID: \"13f59414-113d-4863-88b8-e43c25b17cff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-50e9fc-56cdb5789f-h86nr" Oct 30 18:29:26.891931 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:26.891899 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srcm8\" (UniqueName: \"kubernetes.io/projected/13f59414-113d-4863-88b8-e43c25b17cff-kube-api-access-srcm8\") pod \"payload-pre-processing-e2e-shared-a-w4-50e9fc-56cdb5789f-h86nr\" (UID: \"13f59414-113d-4863-88b8-e43c25b17cff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-50e9fc-56cdb5789f-h86nr" Oct 30 18:29:26.993960 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:26.979701 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz28f\" (UniqueName: \"kubernetes.io/projected/67ec911d-b41b-49f5-b1e3-4637bc7c48f3-kube-api-access-kz28f\") pod \"payload-processing-e2e-shared-a-w4-50e9fc-647b8c7b4-78rvk\" (UID: \"67ec911d-b41b-49f5-b1e3-4637bc7c48f3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-50e9fc-647b8c7b4-78rvk" Oct 30 18:29:26.993960 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:26.979961 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ec911d-b41b-49f5-b1e3-4637bc7c48f3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-50e9fc-647b8c7b4-78rvk\" (UID: \"67ec911d-b41b-49f5-b1e3-4637bc7c48f3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-50e9fc-647b8c7b4-78rvk" Oct 30 18:29:26.993960 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:26.981026 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ec911d-b41b-49f5-b1e3-4637bc7c48f3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-50e9fc-647b8c7b4-78rvk\" (UID: \"67ec911d-b41b-49f5-b1e3-4637bc7c48f3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-50e9fc-647b8c7b4-78rvk" Oct 30 18:29:27.075588 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:27.075487 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz28f\" (UniqueName: \"kubernetes.io/projected/67ec911d-b41b-49f5-b1e3-4637bc7c48f3-kube-api-access-kz28f\") pod \"payload-processing-e2e-shared-a-w4-50e9fc-647b8c7b4-78rvk\" (UID: \"67ec911d-b41b-49f5-b1e3-4637bc7c48f3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-50e9fc-647b8c7b4-78rvk" Oct 30 18:29:27.110355 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:27.103460 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-50e9fc-56cdb5789f-h86nr" Oct 30 18:29:27.116214 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:27.113736 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq"] Oct 30 18:29:27.145873 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:27.145837 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq" Oct 30 18:29:27.166381 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:27.161111 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq"] Oct 30 18:29:27.170373 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:27.169400 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-a2a10a-serving-cert\"" Oct 30 18:29:27.189346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:27.182106 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-50e9fc-647b8c7b4-78rvk" Oct 30 18:29:27.232730 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:27.232684 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fcbeab7-58a0-431a-87e7-e1a4fac339c3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq\" (UID: \"9fcbeab7-58a0-431a-87e7-e1a4fac339c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq" Oct 30 18:29:27.232867 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:27.232794 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zzv5\" (UniqueName: \"kubernetes.io/projected/9fcbeab7-58a0-431a-87e7-e1a4fac339c3-kube-api-access-6zzv5\") pod \"maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq\" (UID: \"9fcbeab7-58a0-431a-87e7-e1a4fac339c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq" Oct 30 18:29:27.232867 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:27.232837 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fcbeab7-58a0-431a-87e7-e1a4fac339c3-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq\" (UID: \"9fcbeab7-58a0-431a-87e7-e1a4fac339c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq" Oct 30 18:29:27.233029 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:27.232881 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fcbeab7-58a0-431a-87e7-e1a4fac339c3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq\" (UID: \"9fcbeab7-58a0-431a-87e7-e1a4fac339c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq" Oct 30 18:29:27.400657 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:27.397755 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fcbeab7-58a0-431a-87e7-e1a4fac339c3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq\" (UID: \"9fcbeab7-58a0-431a-87e7-e1a4fac339c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq" Oct 30 18:29:27.400657 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:27.397860 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zzv5\" (UniqueName: \"kubernetes.io/projected/9fcbeab7-58a0-431a-87e7-e1a4fac339c3-kube-api-access-6zzv5\") pod \"maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq\" (UID: \"9fcbeab7-58a0-431a-87e7-e1a4fac339c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq" Oct 30 18:29:27.400657 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:27.397901 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fcbeab7-58a0-431a-87e7-e1a4fac339c3-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq\" (UID: \"9fcbeab7-58a0-431a-87e7-e1a4fac339c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq" Oct 30 18:29:27.400657 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:27.397944 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fcbeab7-58a0-431a-87e7-e1a4fac339c3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq\" (UID: \"9fcbeab7-58a0-431a-87e7-e1a4fac339c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq" Oct 30 18:29:27.400657 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:27.399052 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fcbeab7-58a0-431a-87e7-e1a4fac339c3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq\" (UID: \"9fcbeab7-58a0-431a-87e7-e1a4fac339c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq" Oct 30 18:29:27.409351 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:27.409206 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fcbeab7-58a0-431a-87e7-e1a4fac339c3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq\" (UID: \"9fcbeab7-58a0-431a-87e7-e1a4fac339c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq" Oct 30 18:29:27.413408 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:27.413375 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fcbeab7-58a0-431a-87e7-e1a4fac339c3-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq\" (UID: \"9fcbeab7-58a0-431a-87e7-e1a4fac339c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq" Oct 30 18:29:27.446377 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:27.441962 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zzv5\" (UniqueName: \"kubernetes.io/projected/9fcbeab7-58a0-431a-87e7-e1a4fac339c3-kube-api-access-6zzv5\") pod \"maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq\" (UID: \"9fcbeab7-58a0-431a-87e7-e1a4fac339c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq" Oct 30 18:29:27.500433 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:27.500397 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq" Oct 30 18:29:27.556905 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:27.556875 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-50e9fc-56cdb5789f-h86nr"] Oct 30 18:29:27.557022 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:29:27.556945 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f59414_113d_4863_88b8_e43c25b17cff.slice/crio-072b6d4bfbc99fe99eb886328ffe2ec2d5c1b23ac89e5c6d6391f782ac19468c WatchSource:0}: Error finding container 072b6d4bfbc99fe99eb886328ffe2ec2d5c1b23ac89e5c6d6391f782ac19468c: Status 404 returned error can't find the container with id 072b6d4bfbc99fe99eb886328ffe2ec2d5c1b23ac89e5c6d6391f782ac19468c Oct 30 18:29:27.590496 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:27.590464 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-50e9fc-647b8c7b4-78rvk"] Oct 30 18:29:27.821415 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:29:27.821381 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fcbeab7_58a0_431a_87e7_e1a4fac339c3.slice/crio-6170b4a47b6c992c639e03049fc4f28e0238c065331ebdbd5f4d03ac9ed6d4ad WatchSource:0}: Error finding container 6170b4a47b6c992c639e03049fc4f28e0238c065331ebdbd5f4d03ac9ed6d4ad: Status 404 returned error can't find the container with id 6170b4a47b6c992c639e03049fc4f28e0238c065331ebdbd5f4d03ac9ed6d4ad Oct 30 18:29:27.822147 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:27.822126 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq"] Oct 30 18:29:27.893422 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:27.893375 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-50e9fc-647b8c7b4-78rvk" event={"ID":"67ec911d-b41b-49f5-b1e3-4637bc7c48f3","Type":"ContainerStarted","Data":"ce2473c9edf9cd1a6177ceaee91cd4eb7a0a8e83daf7d2a046a45957ee245a77"} Oct 30 18:29:27.893905 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:27.893461 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-50e9fc-647b8c7b4-78rvk" Oct 30 18:29:27.893905 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:27.893488 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-50e9fc-647b8c7b4-78rvk" event={"ID":"67ec911d-b41b-49f5-b1e3-4637bc7c48f3","Type":"ContainerStarted","Data":"c8f686f8893110d637ab2e2e74db0cc58afb4c4f6edc49930a050115a51efbf4"} Oct 30 18:29:27.895911 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:27.895869 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-50e9fc-56cdb5789f-h86nr" event={"ID":"13f59414-113d-4863-88b8-e43c25b17cff","Type":"ContainerStarted","Data":"b8f720cdbb798b5e7eaa1831d6d61d8af1e1bbd00066cc9b982cbfa3f6b2b049"} Oct 30 18:29:27.896057 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:27.895932 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-50e9fc-56cdb5789f-h86nr" event={"ID":"13f59414-113d-4863-88b8-e43c25b17cff","Type":"ContainerStarted","Data":"072b6d4bfbc99fe99eb886328ffe2ec2d5c1b23ac89e5c6d6391f782ac19468c"} Oct 30 18:29:27.896136 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:27.896067 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-50e9fc-56cdb5789f-h86nr" Oct 30 18:29:27.897446 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:27.897418 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq" event={"ID":"9fcbeab7-58a0-431a-87e7-e1a4fac339c3","Type":"ContainerStarted","Data":"6170b4a47b6c992c639e03049fc4f28e0238c065331ebdbd5f4d03ac9ed6d4ad"} Oct 30 18:29:27.938413 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:27.938287 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-50e9fc-647b8c7b4-78rvk" podStartSLOduration=1.93826757 podStartE2EDuration="1.93826757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:27.93679901 +0000 UTC m=+2220.424436043" watchObservedRunningTime="2026-10-30 18:29:27.93826757 +0000 UTC m=+2220.425904604" Oct 30 18:29:27.987830 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:27.987778 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-50e9fc-56cdb5789f-h86nr" podStartSLOduration=1.987765156 podStartE2EDuration="1.987765156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:27.985652461 +0000 UTC m=+2220.473289496" watchObservedRunningTime="2026-10-30 18:29:27.987765156 +0000 UTC m=+2220.475402189" Oct 30 18:29:28.918422 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:28.918311 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq" event={"ID":"9fcbeab7-58a0-431a-87e7-e1a4fac339c3","Type":"ContainerStarted","Data":"181d91245914f016c05773b5b66c2423336f6d83f67d6336029aaecb69af4227"} Oct 30 18:29:28.968066 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:28.968003 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq" podStartSLOduration=1.5888743870000002 podStartE2EDuration="1.967984764s" totalImagesPullingTime="379.110377ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:28.96616516 +0000 UTC m=+2221.453802195" watchObservedRunningTime="2026-10-30 18:29:28.967984764 +0000 UTC m=+2221.455621800" Oct 30 18:29:29.964347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:29.962893 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq" Oct 30 18:29:30.388797 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:29:30.388754 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:29:35.308106 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:35.308061 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5cb3d5-558fb66579-zq2b6"] Oct 30 18:29:35.313391 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:35.313360 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5cb3d5-558fb66579-zq2b6" Oct 30 18:29:35.333939 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:35.333913 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5cb3d5-558fb66579-zq2b6"] Oct 30 18:29:35.342144 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:35.342115 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-5cb3d5\"" Oct 30 18:29:35.342362 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:35.342229 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-5cb3d5-dockercfg-fjd26\"" Oct 30 18:29:35.458351 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:35.439481 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd55385-0d97-47c9-9cc8-82659f695458-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-5cb3d5-558fb66579-zq2b6\" (UID: \"2dd55385-0d97-47c9-9cc8-82659f695458\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5cb3d5-558fb66579-zq2b6" Oct 30 18:29:35.458351 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:35.439560 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdpft\" (UniqueName: \"kubernetes.io/projected/2dd55385-0d97-47c9-9cc8-82659f695458-kube-api-access-fdpft\") pod \"payload-pre-processing-e2e-shared-b-w0-5cb3d5-558fb66579-zq2b6\" (UID: \"2dd55385-0d97-47c9-9cc8-82659f695458\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5cb3d5-558fb66579-zq2b6" Oct 30 18:29:35.546973 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:35.546913 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd55385-0d97-47c9-9cc8-82659f695458-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-5cb3d5-558fb66579-zq2b6\" (UID: \"2dd55385-0d97-47c9-9cc8-82659f695458\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5cb3d5-558fb66579-zq2b6" Oct 30 18:29:35.547304 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:35.547279 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdpft\" (UniqueName: \"kubernetes.io/projected/2dd55385-0d97-47c9-9cc8-82659f695458-kube-api-access-fdpft\") pod \"payload-pre-processing-e2e-shared-b-w0-5cb3d5-558fb66579-zq2b6\" (UID: \"2dd55385-0d97-47c9-9cc8-82659f695458\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5cb3d5-558fb66579-zq2b6" Oct 30 18:29:35.549233 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:35.549188 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd55385-0d97-47c9-9cc8-82659f695458-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-5cb3d5-558fb66579-zq2b6\" (UID: \"2dd55385-0d97-47c9-9cc8-82659f695458\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5cb3d5-558fb66579-zq2b6" Oct 30 18:29:35.556744 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:35.556707 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-5cb3d5-c86c6bb98-lb6hc"] Oct 30 18:29:35.597399 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:35.596588 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5cb3d5-c86c6bb98-lb6hc" Oct 30 18:29:35.611393 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:35.603259 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-5cb3d5-c86c6bb98-lb6hc"] Oct 30 18:29:35.623217 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:35.623180 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdpft\" (UniqueName: \"kubernetes.io/projected/2dd55385-0d97-47c9-9cc8-82659f695458-kube-api-access-fdpft\") pod \"payload-pre-processing-e2e-shared-b-w0-5cb3d5-558fb66579-zq2b6\" (UID: \"2dd55385-0d97-47c9-9cc8-82659f695458\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5cb3d5-558fb66579-zq2b6" Oct 30 18:29:35.634849 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:35.634815 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5cb3d5-558fb66579-zq2b6" Oct 30 18:29:35.671965 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:35.671932 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4hs2\" (UniqueName: \"kubernetes.io/projected/abd1468f-513d-4fcb-8050-1b60650f28a6-kube-api-access-j4hs2\") pod \"payload-processing-e2e-shared-b-w0-5cb3d5-c86c6bb98-lb6hc\" (UID: \"abd1468f-513d-4fcb-8050-1b60650f28a6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5cb3d5-c86c6bb98-lb6hc" Oct 30 18:29:35.672216 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:35.672191 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd1468f-513d-4fcb-8050-1b60650f28a6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-5cb3d5-c86c6bb98-lb6hc\" (UID: \"abd1468f-513d-4fcb-8050-1b60650f28a6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5cb3d5-c86c6bb98-lb6hc" Oct 30 18:29:35.774512 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:35.774449 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4hs2\" (UniqueName: \"kubernetes.io/projected/abd1468f-513d-4fcb-8050-1b60650f28a6-kube-api-access-j4hs2\") pod \"payload-processing-e2e-shared-b-w0-5cb3d5-c86c6bb98-lb6hc\" (UID: \"abd1468f-513d-4fcb-8050-1b60650f28a6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5cb3d5-c86c6bb98-lb6hc" Oct 30 18:29:35.774743 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:35.774515 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd1468f-513d-4fcb-8050-1b60650f28a6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-5cb3d5-c86c6bb98-lb6hc\" (UID: \"abd1468f-513d-4fcb-8050-1b60650f28a6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5cb3d5-c86c6bb98-lb6hc" Oct 30 18:29:35.775633 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:35.775603 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd1468f-513d-4fcb-8050-1b60650f28a6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-5cb3d5-c86c6bb98-lb6hc\" (UID: \"abd1468f-513d-4fcb-8050-1b60650f28a6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5cb3d5-c86c6bb98-lb6hc" Oct 30 18:29:35.803076 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:35.802969 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4hs2\" (UniqueName: \"kubernetes.io/projected/abd1468f-513d-4fcb-8050-1b60650f28a6-kube-api-access-j4hs2\") pod \"payload-processing-e2e-shared-b-w0-5cb3d5-c86c6bb98-lb6hc\" (UID: \"abd1468f-513d-4fcb-8050-1b60650f28a6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5cb3d5-c86c6bb98-lb6hc" Oct 30 18:29:35.935042 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:35.934937 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5cb3d5-c86c6bb98-lb6hc" Oct 30 18:29:35.980937 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:35.980907 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5cb3d5-558fb66579-zq2b6"] Oct 30 18:29:35.983114 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:29:35.983065 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd55385_0d97_47c9_9cc8_82659f695458.slice/crio-93574f55e506747c761d4d02b91016aa3dce9436e4a1148c84a24dba67b1fa13 WatchSource:0}: Error finding container 93574f55e506747c761d4d02b91016aa3dce9436e4a1148c84a24dba67b1fa13: Status 404 returned error can't find the container with id 93574f55e506747c761d4d02b91016aa3dce9436e4a1148c84a24dba67b1fa13 Oct 30 18:29:35.990505 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:35.990482 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq" Oct 30 18:29:36.037422 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:36.037346 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5cb3d5-558fb66579-zq2b6" event={"ID":"2dd55385-0d97-47c9-9cc8-82659f695458","Type":"ContainerStarted","Data":"93574f55e506747c761d4d02b91016aa3dce9436e4a1148c84a24dba67b1fa13"} Oct 30 18:29:36.256686 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:36.256653 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-5cb3d5-c86c6bb98-lb6hc"] Oct 30 18:29:36.256686 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:29:36.256675 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd1468f_513d_4fcb_8050_1b60650f28a6.slice/crio-af930a29b4b9ba99ae748aa9032d268f948b5cdd7c34b07f97ca39e3cd16a22e WatchSource:0}: Error finding container af930a29b4b9ba99ae748aa9032d268f948b5cdd7c34b07f97ca39e3cd16a22e: Status 404 returned error can't find the container with id af930a29b4b9ba99ae748aa9032d268f948b5cdd7c34b07f97ca39e3cd16a22e Oct 30 18:29:36.892106 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:36.892067 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e77b1d-9fdccb855-cfxfl" Oct 30 18:29:36.893680 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:36.893621 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e77b1d-55ddb87958-l9wgq" Oct 30 18:29:37.046210 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:37.046171 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5cb3d5-c86c6bb98-lb6hc" event={"ID":"abd1468f-513d-4fcb-8050-1b60650f28a6","Type":"ContainerStarted","Data":"0f07d3b5fc074cfaef897696e6eba666a43918e2d7a60207657617304785d6c3"} Oct 30 18:29:37.046630 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:37.046601 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5cb3d5-c86c6bb98-lb6hc" Oct 30 18:29:37.046779 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:37.046648 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5cb3d5-c86c6bb98-lb6hc" event={"ID":"abd1468f-513d-4fcb-8050-1b60650f28a6","Type":"ContainerStarted","Data":"af930a29b4b9ba99ae748aa9032d268f948b5cdd7c34b07f97ca39e3cd16a22e"} Oct 30 18:29:37.048290 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:37.048262 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5cb3d5-558fb66579-zq2b6" event={"ID":"2dd55385-0d97-47c9-9cc8-82659f695458","Type":"ContainerStarted","Data":"79bde611ee704de054b481446d7ca01f0c5bf796d1ec95659cca7ebbc4ec9136"} Oct 30 18:29:37.048435 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:37.048356 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5cb3d5-558fb66579-zq2b6" Oct 30 18:29:37.106555 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:37.106486 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5cb3d5-c86c6bb98-lb6hc" podStartSLOduration=2.10646449 podStartE2EDuration="2.10646449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:37.090537366 +0000 UTC m=+2229.578174400" watchObservedRunningTime="2026-10-30 18:29:37.10646449 +0000 UTC m=+2229.594101525" Oct 30 18:29:37.148150 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:37.148034 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5cb3d5-558fb66579-zq2b6" podStartSLOduration=2.148020928 podStartE2EDuration="2.148020928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:37.14489502 +0000 UTC m=+2229.632532054" watchObservedRunningTime="2026-10-30 18:29:37.148020928 +0000 UTC m=+2229.635657962" Oct 30 18:29:38.922232 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:38.922198 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-50e9fc-647b8c7b4-78rvk" Oct 30 18:29:38.922715 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:38.922278 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-50e9fc-56cdb5789f-h86nr" Oct 30 18:29:40.695662 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:40.695626 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6"] Oct 30 18:29:40.700651 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:40.700629 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6" Oct 30 18:29:40.722742 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:40.722710 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6"] Oct 30 18:29:40.731805 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:40.731771 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-e77b1d-serving-cert\"" Oct 30 18:29:40.875060 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:40.875024 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe742589-45cc-4159-b8ba-b4402856f2db-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6\" (UID: \"fe742589-45cc-4159-b8ba-b4402856f2db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6" Oct 30 18:29:40.875272 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:40.875076 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe742589-45cc-4159-b8ba-b4402856f2db-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6\" (UID: \"fe742589-45cc-4159-b8ba-b4402856f2db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6" Oct 30 18:29:40.875272 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:40.875249 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe742589-45cc-4159-b8ba-b4402856f2db-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6\" (UID: \"fe742589-45cc-4159-b8ba-b4402856f2db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6" Oct 30 18:29:40.875627 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:40.875601 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2znqm\" (UniqueName: \"kubernetes.io/projected/fe742589-45cc-4159-b8ba-b4402856f2db-kube-api-access-2znqm\") pod \"maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6\" (UID: \"fe742589-45cc-4159-b8ba-b4402856f2db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6" Oct 30 18:29:40.979052 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:40.978998 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe742589-45cc-4159-b8ba-b4402856f2db-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6\" (UID: \"fe742589-45cc-4159-b8ba-b4402856f2db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6" Oct 30 18:29:40.979316 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:40.979141 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe742589-45cc-4159-b8ba-b4402856f2db-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6\" (UID: \"fe742589-45cc-4159-b8ba-b4402856f2db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6" Oct 30 18:29:41.017771 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:41.001026 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2znqm\" (UniqueName: \"kubernetes.io/projected/fe742589-45cc-4159-b8ba-b4402856f2db-kube-api-access-2znqm\") pod \"maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6\" (UID: \"fe742589-45cc-4159-b8ba-b4402856f2db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6" Oct 30 18:29:41.017771 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:41.001152 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe742589-45cc-4159-b8ba-b4402856f2db-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6\" (UID: \"fe742589-45cc-4159-b8ba-b4402856f2db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6" Oct 30 18:29:41.017771 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:41.003207 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe742589-45cc-4159-b8ba-b4402856f2db-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6\" (UID: \"fe742589-45cc-4159-b8ba-b4402856f2db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6" Oct 30 18:29:41.017771 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:41.005247 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe742589-45cc-4159-b8ba-b4402856f2db-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6\" (UID: \"fe742589-45cc-4159-b8ba-b4402856f2db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6" Oct 30 18:29:41.019214 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:41.019181 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe742589-45cc-4159-b8ba-b4402856f2db-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6\" (UID: \"fe742589-45cc-4159-b8ba-b4402856f2db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6" Oct 30 18:29:41.067783 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:41.064306 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2znqm\" (UniqueName: \"kubernetes.io/projected/fe742589-45cc-4159-b8ba-b4402856f2db-kube-api-access-2znqm\") pod \"maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6\" (UID: \"fe742589-45cc-4159-b8ba-b4402856f2db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6" Oct 30 18:29:41.317695 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:41.317605 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6" Oct 30 18:29:41.644963 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:41.644931 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6"] Oct 30 18:29:42.099851 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:42.099815 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6" event={"ID":"fe742589-45cc-4159-b8ba-b4402856f2db","Type":"ContainerStarted","Data":"4f78a05c3de3a84bc14563c87b8dbf23917483251bd7afa6a7befd3a28f3d15c"} Oct 30 18:29:42.395618 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:42.395535 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg"] Oct 30 18:29:42.400462 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:42.400440 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg" Oct 30 18:29:42.423379 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:42.422129 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-50e9fc-serving-cert\"" Oct 30 18:29:42.427525 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:42.427495 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg"] Oct 30 18:29:42.469980 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:42.469946 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4deaf1e2-755d-41bb-b3b7-ff08a88186d0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg\" (UID: \"4deaf1e2-755d-41bb-b3b7-ff08a88186d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg" Oct 30 18:29:42.470253 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:42.470233 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4deaf1e2-755d-41bb-b3b7-ff08a88186d0-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg\" (UID: \"4deaf1e2-755d-41bb-b3b7-ff08a88186d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg" Oct 30 18:29:42.470418 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:42.470394 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4v5x\" (UniqueName: \"kubernetes.io/projected/4deaf1e2-755d-41bb-b3b7-ff08a88186d0-kube-api-access-b4v5x\") pod \"maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg\" (UID: \"4deaf1e2-755d-41bb-b3b7-ff08a88186d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg" Oct 30 18:29:42.470466 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:42.470435 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4deaf1e2-755d-41bb-b3b7-ff08a88186d0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg\" (UID: \"4deaf1e2-755d-41bb-b3b7-ff08a88186d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg" Oct 30 18:29:42.572473 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:42.572439 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4deaf1e2-755d-41bb-b3b7-ff08a88186d0-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg\" (UID: \"4deaf1e2-755d-41bb-b3b7-ff08a88186d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg" Oct 30 18:29:42.572669 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:42.572579 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4v5x\" (UniqueName: \"kubernetes.io/projected/4deaf1e2-755d-41bb-b3b7-ff08a88186d0-kube-api-access-b4v5x\") pod \"maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg\" (UID: \"4deaf1e2-755d-41bb-b3b7-ff08a88186d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg" Oct 30 18:29:42.572669 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:42.572617 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4deaf1e2-755d-41bb-b3b7-ff08a88186d0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg\" (UID: \"4deaf1e2-755d-41bb-b3b7-ff08a88186d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg" Oct 30 18:29:42.572847 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:42.572692 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4deaf1e2-755d-41bb-b3b7-ff08a88186d0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg\" (UID: \"4deaf1e2-755d-41bb-b3b7-ff08a88186d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg" Oct 30 18:29:42.573943 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:42.573916 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4deaf1e2-755d-41bb-b3b7-ff08a88186d0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg\" (UID: \"4deaf1e2-755d-41bb-b3b7-ff08a88186d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg" Oct 30 18:29:42.577642 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:42.576737 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4deaf1e2-755d-41bb-b3b7-ff08a88186d0-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg\" (UID: \"4deaf1e2-755d-41bb-b3b7-ff08a88186d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg" Oct 30 18:29:42.591760 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:42.591727 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4deaf1e2-755d-41bb-b3b7-ff08a88186d0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg\" (UID: \"4deaf1e2-755d-41bb-b3b7-ff08a88186d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg" Oct 30 18:29:42.631957 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:42.627857 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4v5x\" (UniqueName: \"kubernetes.io/projected/4deaf1e2-755d-41bb-b3b7-ff08a88186d0-kube-api-access-b4v5x\") pod \"maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg\" (UID: \"4deaf1e2-755d-41bb-b3b7-ff08a88186d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg" Oct 30 18:29:42.724660 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:42.724610 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg" Oct 30 18:29:43.054007 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:43.053974 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg"] Oct 30 18:29:43.054147 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:29:43.053986 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4deaf1e2_755d_41bb_b3b7_ff08a88186d0.slice/crio-1f454d2cfeb70acf4b0049ca3187593866cb3882a8ea18c4332b9696e0e028af WatchSource:0}: Error finding container 1f454d2cfeb70acf4b0049ca3187593866cb3882a8ea18c4332b9696e0e028af: Status 404 returned error can't find the container with id 1f454d2cfeb70acf4b0049ca3187593866cb3882a8ea18c4332b9696e0e028af Oct 30 18:29:43.110390 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:43.110348 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg" event={"ID":"4deaf1e2-755d-41bb-b3b7-ff08a88186d0","Type":"ContainerStarted","Data":"1f454d2cfeb70acf4b0049ca3187593866cb3882a8ea18c4332b9696e0e028af"} Oct 30 18:29:43.112425 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:43.112385 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6" event={"ID":"fe742589-45cc-4159-b8ba-b4402856f2db","Type":"ContainerStarted","Data":"f48aa5cf9f96078a88051f6cf13f60ae1f1a795d1404e4999e5e1acd04e53705"} Oct 30 18:29:43.112594 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:43.112546 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6" Oct 30 18:29:43.162786 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:43.162732 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6" podStartSLOduration=2.747127161 podStartE2EDuration="3.162716202s" totalImagesPullingTime="415.589041ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:43.15925582 +0000 UTC m=+2235.646892855" watchObservedRunningTime="2026-10-30 18:29:43.162716202 +0000 UTC m=+2235.650353235" Oct 30 18:29:44.122098 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:44.121996 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg" event={"ID":"4deaf1e2-755d-41bb-b3b7-ff08a88186d0","Type":"ContainerStarted","Data":"9e8dbac784cd68944e95fa0823f47667690ca446f857d451557058810923604c"} Oct 30 18:29:44.122612 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:44.122098 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg" Oct 30 18:29:44.173960 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:44.173908 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg" podStartSLOduration=1.7610905780000001 podStartE2EDuration="2.173893956s" totalImagesPullingTime="412.803378ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:44.168490881 +0000 UTC m=+2236.656127916" watchObservedRunningTime="2026-10-30 18:29:44.173893956 +0000 UTC m=+2236.661530989" Oct 30 18:29:48.059264 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:48.059233 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5cb3d5-558fb66579-zq2b6" Oct 30 18:29:48.059782 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:48.059316 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5cb3d5-c86c6bb98-lb6hc" Oct 30 18:29:49.128195 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:49.128167 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6" Oct 30 18:29:50.136295 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:50.136262 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg" Oct 30 18:29:54.903745 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:54.903709 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m"] Oct 30 18:29:54.908947 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:54.908924 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m" Oct 30 18:29:54.940967 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:54.940931 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-5cb3d5-serving-cert\"" Oct 30 18:29:54.947171 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:54.947134 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m"] Oct 30 18:29:55.009143 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:55.009111 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9a7536b-d288-4ea3-b1d1-f824ec859a47-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m\" (UID: \"b9a7536b-d288-4ea3-b1d1-f824ec859a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m" Oct 30 18:29:55.009432 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:55.009404 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh8zx\" (UniqueName: \"kubernetes.io/projected/b9a7536b-d288-4ea3-b1d1-f824ec859a47-kube-api-access-zh8zx\") pod \"maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m\" (UID: \"b9a7536b-d288-4ea3-b1d1-f824ec859a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m" Oct 30 18:29:55.009540 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:55.009519 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9a7536b-d288-4ea3-b1d1-f824ec859a47-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m\" (UID: \"b9a7536b-d288-4ea3-b1d1-f824ec859a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m" Oct 30 18:29:55.009631 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:55.009616 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9a7536b-d288-4ea3-b1d1-f824ec859a47-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m\" (UID: \"b9a7536b-d288-4ea3-b1d1-f824ec859a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m" Oct 30 18:29:55.129602 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:55.115760 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh8zx\" (UniqueName: \"kubernetes.io/projected/b9a7536b-d288-4ea3-b1d1-f824ec859a47-kube-api-access-zh8zx\") pod \"maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m\" (UID: \"b9a7536b-d288-4ea3-b1d1-f824ec859a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m" Oct 30 18:29:55.131958 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:55.131919 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9a7536b-d288-4ea3-b1d1-f824ec859a47-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m\" (UID: \"b9a7536b-d288-4ea3-b1d1-f824ec859a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m" Oct 30 18:29:55.132284 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:55.132255 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9a7536b-d288-4ea3-b1d1-f824ec859a47-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m\" (UID: \"b9a7536b-d288-4ea3-b1d1-f824ec859a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m" Oct 30 18:29:55.132762 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:55.132738 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9a7536b-d288-4ea3-b1d1-f824ec859a47-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m\" (UID: \"b9a7536b-d288-4ea3-b1d1-f824ec859a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m" Oct 30 18:29:55.134428 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:29:55.134393 4455 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5cb3d5-serving-cert: secret "maas-api-e2e-shared-b-w0-5cb3d5-serving-cert" not found Oct 30 18:29:55.134685 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:29:55.134672 4455 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9a7536b-d288-4ea3-b1d1-f824ec859a47-maas-api-tls podName:b9a7536b-d288-4ea3-b1d1-f824ec859a47 nodeName:}" failed. No retries permitted until 2026-10-30 18:29:55.634632932 +0000 UTC m=+2248.122269959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b9a7536b-d288-4ea3-b1d1-f824ec859a47-maas-api-tls") pod "maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m" (UID: "b9a7536b-d288-4ea3-b1d1-f824ec859a47") : secret "maas-api-e2e-shared-b-w0-5cb3d5-serving-cert" not found Oct 30 18:29:55.160916 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:55.160757 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9a7536b-d288-4ea3-b1d1-f824ec859a47-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m\" (UID: \"b9a7536b-d288-4ea3-b1d1-f824ec859a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m" Oct 30 18:29:55.171830 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:55.171737 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh8zx\" (UniqueName: \"kubernetes.io/projected/b9a7536b-d288-4ea3-b1d1-f824ec859a47-kube-api-access-zh8zx\") pod \"maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m\" (UID: \"b9a7536b-d288-4ea3-b1d1-f824ec859a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m" Oct 30 18:29:55.173750 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:55.173724 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9a7536b-d288-4ea3-b1d1-f824ec859a47-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m\" (UID: \"b9a7536b-d288-4ea3-b1d1-f824ec859a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m" Oct 30 18:29:55.641288 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:55.641244 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9a7536b-d288-4ea3-b1d1-f824ec859a47-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m\" (UID: \"b9a7536b-d288-4ea3-b1d1-f824ec859a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m" Oct 30 18:29:55.644158 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:55.644090 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9a7536b-d288-4ea3-b1d1-f824ec859a47-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m\" (UID: \"b9a7536b-d288-4ea3-b1d1-f824ec859a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m" Oct 30 18:29:55.824704 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:55.824663 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m" Oct 30 18:29:56.302148 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:56.302104 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m"] Oct 30 18:29:57.315525 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:57.315483 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m" event={"ID":"b9a7536b-d288-4ea3-b1d1-f824ec859a47","Type":"ContainerStarted","Data":"c15d1d7108bd1881634d5fd0f048e1ff9131f51d65cdd24e710785d22c9813f3"} Oct 30 18:29:57.316013 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:57.315553 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m" event={"ID":"b9a7536b-d288-4ea3-b1d1-f824ec859a47","Type":"ContainerStarted","Data":"a9cd86cd260ad9a9b917b30f9213dfac7c5a4cf9cce71a11bda58b9882ccc807"} Oct 30 18:29:57.316013 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:57.315796 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m" Oct 30 18:29:57.365268 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:29:57.365198 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m" podStartSLOduration=2.880064788 podStartE2EDuration="3.365177628s" totalImagesPullingTime="485.11284ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:57.360776996 +0000 UTC m=+2249.848414034" watchObservedRunningTime="2026-10-30 18:29:57.365177628 +0000 UTC m=+2249.852814663" Oct 30 18:30:00.672346 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:30:00.670107 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:30:02.159350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:02.158455 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:30:03.176351 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:03.159436 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:03.206347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:03.178572 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:30:03.851379 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:03.848545 4455 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:30:03.851379 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:03.848634 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" podUID="e972725e-489d-4f2a-a734-27d7eb570858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:04.015742 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:04.015706 4455 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:30:04.015992 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:04.015967 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" podUID="ac80ccea-a611-4de1-951c-479ad3d827a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:04.493652 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:04.492430 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" podUID="25474977-dd58-4ead-9910-f20df0701ae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.184:8000/ready\": context deadline exceeded" Oct 30 18:30:04.519388 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:04.500653 4455 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:30:04.519388 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:04.500719 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" podUID="93b710b6-b5a8-4fe0-ac70-861dd036b70d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:04.519388 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:04.502158 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m" Oct 30 18:30:04.574187 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:04.574137 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:30:05.497358 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:05.496072 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" podUID="25474977-dd58-4ead-9910-f20df0701ae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.184:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:05.786399 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:05.782157 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:05.844694 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:05.820260 4455 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:30:05.844694 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:05.820340 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" podUID="1d98345b-38d7-49b3-8092-a41b1f584fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:06.196408 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:06.193293 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" Oct 30 18:30:06.740985 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:06.740210 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" podUID="25474977-dd58-4ead-9910-f20df0701ae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.184:8000/ready\": context deadline exceeded" Oct 30 18:30:06.823350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:06.797874 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:30:06.910481 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:06.910439 4455 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:30:06.910798 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:06.910767 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" podUID="7c9b7598-cec3-43ab-90cf-93e7b407d68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:07.371351 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:07.351194 4455 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/health\": context deadline exceeded" Oct 30 18:30:07.765354 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:07.750433 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" podUID="25474977-dd58-4ead-9910-f20df0701ae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.184:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:07.862348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:07.838394 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:08.162025 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:08.161930 4455 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:30:08.162292 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:08.162260 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" podUID="bbf4bc50-3670-40de-8159-3d8805fdd601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:08.343260 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:08.343210 4455 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" podUID="25474977-dd58-4ead-9910-f20df0701ae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.184:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:08.404382 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:08.404306 4455 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:30:08.404606 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:08.404413 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" podUID="3daa8589-9def-486f-82ee-0c923a18b68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:08.848727 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:08.848628 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" podUID="25474977-dd58-4ead-9910-f20df0701ae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.184:8000/ready\": context deadline exceeded" Oct 30 18:30:08.888002 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:08.886644 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:30:09.690817 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:09.690782 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:30:09.692127 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:09.691160 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" Oct 30 18:30:20.398697 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:20.398661 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m"] Oct 30 18:30:20.405886 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:20.405865 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" Oct 30 18:30:20.434867 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:20.434829 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m"] Oct 30 18:30:20.481017 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:20.480811 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72dfe139-2b2a-4787-b415-6bee9768d82c-workload-socket\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" Oct 30 18:30:20.481017 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:20.480860 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72dfe139-2b2a-4787-b415-6bee9768d82c-istio-token\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" Oct 30 18:30:20.481017 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:20.480913 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72dfe139-2b2a-4787-b415-6bee9768d82c-istiod-ca-cert\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" Oct 30 18:30:20.481017 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:20.480939 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6kx2\" (UniqueName: \"kubernetes.io/projected/72dfe139-2b2a-4787-b415-6bee9768d82c-kube-api-access-t6kx2\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" Oct 30 18:30:20.482101 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:20.481067 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72dfe139-2b2a-4787-b415-6bee9768d82c-istio-podinfo\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" Oct 30 18:30:20.482101 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:20.481143 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72dfe139-2b2a-4787-b415-6bee9768d82c-istio-data\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" Oct 30 18:30:20.482101 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:20.481204 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72dfe139-2b2a-4787-b415-6bee9768d82c-credential-socket\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" Oct 30 18:30:20.482101 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:20.481244 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72dfe139-2b2a-4787-b415-6bee9768d82c-istio-envoy\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" Oct 30 18:30:20.482101 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:20.481292 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72dfe139-2b2a-4787-b415-6bee9768d82c-workload-certs\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" Oct 30 18:30:20.504038 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:20.504010 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-a2a10a-openshift-default-dockercfg-4fw27\"" Oct 30 18:30:20.583277 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:20.583242 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72dfe139-2b2a-4787-b415-6bee9768d82c-istio-podinfo\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" Oct 30 18:30:20.583486 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:20.583308 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72dfe139-2b2a-4787-b415-6bee9768d82c-istio-data\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" Oct 30 18:30:20.583486 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:20.583408 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72dfe139-2b2a-4787-b415-6bee9768d82c-credential-socket\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" Oct 30 18:30:20.583486 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:20.583449 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72dfe139-2b2a-4787-b415-6bee9768d82c-istio-envoy\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" Oct 30 18:30:20.583679 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:20.583500 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72dfe139-2b2a-4787-b415-6bee9768d82c-workload-certs\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" Oct 30 18:30:20.583679 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:20.583670 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72dfe139-2b2a-4787-b415-6bee9768d82c-workload-socket\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" Oct 30 18:30:20.583805 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:20.583698 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72dfe139-2b2a-4787-b415-6bee9768d82c-istio-token\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" Oct 30 18:30:20.583805 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:20.583745 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72dfe139-2b2a-4787-b415-6bee9768d82c-istiod-ca-cert\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" Oct 30 18:30:20.583805 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:20.583771 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6kx2\" (UniqueName: \"kubernetes.io/projected/72dfe139-2b2a-4787-b415-6bee9768d82c-kube-api-access-t6kx2\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" Oct 30 18:30:20.583968 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:20.583828 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72dfe139-2b2a-4787-b415-6bee9768d82c-credential-socket\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" Oct 30 18:30:20.584034 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:20.583978 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72dfe139-2b2a-4787-b415-6bee9768d82c-workload-certs\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" Oct 30 18:30:20.584034 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:20.584004 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72dfe139-2b2a-4787-b415-6bee9768d82c-istio-data\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" Oct 30 18:30:20.584606 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:20.584586 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72dfe139-2b2a-4787-b415-6bee9768d82c-workload-socket\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" Oct 30 18:30:20.584701 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:20.584672 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72dfe139-2b2a-4787-b415-6bee9768d82c-istiod-ca-cert\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" Oct 30 18:30:20.586291 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:20.586266 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72dfe139-2b2a-4787-b415-6bee9768d82c-istio-podinfo\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" Oct 30 18:30:20.586625 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:20.586602 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72dfe139-2b2a-4787-b415-6bee9768d82c-istio-envoy\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" Oct 30 18:30:20.609626 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:20.609599 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72dfe139-2b2a-4787-b415-6bee9768d82c-istio-token\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" Oct 30 18:30:20.609747 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:20.609645 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6kx2\" (UniqueName: \"kubernetes.io/projected/72dfe139-2b2a-4787-b415-6bee9768d82c-kube-api-access-t6kx2\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" Oct 30 18:30:20.731436 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:20.731399 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" Oct 30 18:30:21.064892 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:21.064862 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m"] Oct 30 18:30:21.065569 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:30:21.065536 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72dfe139_2b2a_4787_b415_6bee9768d82c.slice/crio-621e57bd13d5cd50c6162c3e4272be7822fdea77b149856094cecfab800a820c WatchSource:0}: Error finding container 621e57bd13d5cd50c6162c3e4272be7822fdea77b149856094cecfab800a820c: Status 404 returned error can't find the container with id 621e57bd13d5cd50c6162c3e4272be7822fdea77b149856094cecfab800a820c Oct 30 18:30:21.068826 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:21.068786 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:30:21.068923 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:21.068874 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:30:21.068988 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:21.068920 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:30:21.140838 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:21.140808 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" event={"ID":"72dfe139-2b2a-4787-b415-6bee9768d82c","Type":"ContainerStarted","Data":"621e57bd13d5cd50c6162c3e4272be7822fdea77b149856094cecfab800a820c"} Oct 30 18:30:22.150587 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:22.150545 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" event={"ID":"72dfe139-2b2a-4787-b415-6bee9768d82c","Type":"ContainerStarted","Data":"f8eedc22da04b9d828b7f3dd35edb3b15be631c6ec6857a2a22b9fca07898a45"} Oct 30 18:30:22.241116 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:22.241060 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" podStartSLOduration=2.241046297 podStartE2EDuration="2.241046297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:22.223198322 +0000 UTC m=+2274.710835357" watchObservedRunningTime="2026-10-30 18:30:22.241046297 +0000 UTC m=+2274.728683330" Oct 30 18:30:22.731831 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:22.731799 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" Oct 30 18:30:22.733304 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:22.733281 4455 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 30 18:30:22.733440 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:22.733361 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" podUID="72dfe139-2b2a-4787-b415-6bee9768d82c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 30 18:30:23.739378 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:23.734748 4455 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 30 18:30:23.739378 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:23.734815 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" podUID="72dfe139-2b2a-4787-b415-6bee9768d82c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 30 18:30:24.731821 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:24.731782 4455 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 30 18:30:24.732087 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:24.732057 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" podUID="72dfe139-2b2a-4787-b415-6bee9768d82c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 30 18:30:25.731817 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:25.731775 4455 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 30 18:30:25.732420 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:25.731857 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" podUID="72dfe139-2b2a-4787-b415-6bee9768d82c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 30 18:30:26.731999 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:26.731962 4455 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 30 18:30:26.732503 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:26.732015 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" podUID="72dfe139-2b2a-4787-b415-6bee9768d82c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 30 18:30:27.802001 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:27.801967 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" Oct 30 18:30:27.802682 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:27.802657 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" Oct 30 18:30:27.808125 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:27.808094 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" Oct 30 18:30:30.210015 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:30.209977 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w"] Oct 30 18:30:30.218602 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:30.218571 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w" Oct 30 18:30:30.255576 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:30.255537 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-b00d69-serving-cert\"" Oct 30 18:30:30.258379 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:30.258305 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w"] Oct 30 18:30:30.289877 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:30.289838 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6058153-6f0e-4d2f-ac5c-53be26e376be-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w\" (UID: \"d6058153-6f0e-4d2f-ac5c-53be26e376be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w" Oct 30 18:30:30.290087 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:30.290046 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6058153-6f0e-4d2f-ac5c-53be26e376be-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w\" (UID: \"d6058153-6f0e-4d2f-ac5c-53be26e376be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w" Oct 30 18:30:30.290213 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:30.290164 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6058153-6f0e-4d2f-ac5c-53be26e376be-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w\" (UID: \"d6058153-6f0e-4d2f-ac5c-53be26e376be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w" Oct 30 18:30:30.290444 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:30.290416 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqf56\" (UniqueName: \"kubernetes.io/projected/d6058153-6f0e-4d2f-ac5c-53be26e376be-kube-api-access-kqf56\") pod \"maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w\" (UID: \"d6058153-6f0e-4d2f-ac5c-53be26e376be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w" Oct 30 18:30:30.391948 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:30.391909 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqf56\" (UniqueName: \"kubernetes.io/projected/d6058153-6f0e-4d2f-ac5c-53be26e376be-kube-api-access-kqf56\") pod \"maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w\" (UID: \"d6058153-6f0e-4d2f-ac5c-53be26e376be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w" Oct 30 18:30:30.392362 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:30.392344 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6058153-6f0e-4d2f-ac5c-53be26e376be-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w\" (UID: \"d6058153-6f0e-4d2f-ac5c-53be26e376be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w" Oct 30 18:30:30.392615 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:30.392589 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6058153-6f0e-4d2f-ac5c-53be26e376be-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w\" (UID: \"d6058153-6f0e-4d2f-ac5c-53be26e376be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w" Oct 30 18:30:30.392779 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:30:30.392756 4455 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b00d69-serving-cert: secret "maas-api-e2e-shared-b-w2-b00d69-serving-cert" not found Oct 30 18:30:30.392865 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:30.392852 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6058153-6f0e-4d2f-ac5c-53be26e376be-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w\" (UID: \"d6058153-6f0e-4d2f-ac5c-53be26e376be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w" Oct 30 18:30:30.393300 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:30:30.392871 4455 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6058153-6f0e-4d2f-ac5c-53be26e376be-maas-api-tls podName:d6058153-6f0e-4d2f-ac5c-53be26e376be nodeName:}" failed. No retries permitted until 2026-10-30 18:30:30.892849964 +0000 UTC m=+2283.380486976 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d6058153-6f0e-4d2f-ac5c-53be26e376be-maas-api-tls") pod "maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w" (UID: "d6058153-6f0e-4d2f-ac5c-53be26e376be") : secret "maas-api-e2e-shared-b-w2-b00d69-serving-cert" not found Oct 30 18:30:30.393580 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:30.393039 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6058153-6f0e-4d2f-ac5c-53be26e376be-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w\" (UID: \"d6058153-6f0e-4d2f-ac5c-53be26e376be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w" Oct 30 18:30:30.395635 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:30.395603 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6058153-6f0e-4d2f-ac5c-53be26e376be-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w\" (UID: \"d6058153-6f0e-4d2f-ac5c-53be26e376be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w" Oct 30 18:30:30.442348 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:30:30.423836 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:30:30.547449 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:30.540716 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqf56\" (UniqueName: \"kubernetes.io/projected/d6058153-6f0e-4d2f-ac5c-53be26e376be-kube-api-access-kqf56\") pod \"maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w\" (UID: \"d6058153-6f0e-4d2f-ac5c-53be26e376be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w" Oct 30 18:30:30.926291 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:30.926202 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6058153-6f0e-4d2f-ac5c-53be26e376be-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w\" (UID: \"d6058153-6f0e-4d2f-ac5c-53be26e376be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w" Oct 30 18:30:30.929019 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:30.928992 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6058153-6f0e-4d2f-ac5c-53be26e376be-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w\" (UID: \"d6058153-6f0e-4d2f-ac5c-53be26e376be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w" Oct 30 18:30:31.136357 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:31.136278 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w" Oct 30 18:30:31.480785 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:31.480749 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w"] Oct 30 18:30:31.483796 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:30:31.483766 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6058153_6f0e_4d2f_ac5c_53be26e376be.slice/crio-653b29ecbf475babbc722b7cc02891ba28d8f6fa374fceff6b09d6775834128d WatchSource:0}: Error finding container 653b29ecbf475babbc722b7cc02891ba28d8f6fa374fceff6b09d6775834128d: Status 404 returned error can't find the container with id 653b29ecbf475babbc722b7cc02891ba28d8f6fa374fceff6b09d6775834128d Oct 30 18:30:32.324023 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:32.323907 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w" event={"ID":"d6058153-6f0e-4d2f-ac5c-53be26e376be","Type":"ContainerStarted","Data":"d6102c4863b50757fb6eab37a974772adbb6d3bd59d94da86b94dca2f4d680e9"} Oct 30 18:30:32.324023 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:32.323982 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w" event={"ID":"d6058153-6f0e-4d2f-ac5c-53be26e376be","Type":"ContainerStarted","Data":"653b29ecbf475babbc722b7cc02891ba28d8f6fa374fceff6b09d6775834128d"} Oct 30 18:30:32.324219 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:32.324030 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w" Oct 30 18:30:32.371757 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:32.371700 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w" podStartSLOduration=1.852642976 podStartE2EDuration="2.371685696s" totalImagesPullingTime="519.04272ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:32.370760519 +0000 UTC m=+2284.858397554" watchObservedRunningTime="2026-10-30 18:30:32.371685696 +0000 UTC m=+2284.859322730" Oct 30 18:30:38.396516 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:38.396483 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w" Oct 30 18:30:54.273501 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:54.273469 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n"] Oct 30 18:30:54.282415 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:54.281219 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n" Oct 30 18:30:54.305435 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:54.305398 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-c8eef3-serving-cert\"" Oct 30 18:30:54.326946 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:54.326906 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n"] Oct 30 18:30:54.382214 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:54.382161 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a11771b6-cb92-4783-bd70-4eb964d94f00-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n\" (UID: \"a11771b6-cb92-4783-bd70-4eb964d94f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n" Oct 30 18:30:54.382438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:54.382221 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p77tf\" (UniqueName: \"kubernetes.io/projected/a11771b6-cb92-4783-bd70-4eb964d94f00-kube-api-access-p77tf\") pod \"maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n\" (UID: \"a11771b6-cb92-4783-bd70-4eb964d94f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n" Oct 30 18:30:54.382597 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:54.382577 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a11771b6-cb92-4783-bd70-4eb964d94f00-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n\" (UID: \"a11771b6-cb92-4783-bd70-4eb964d94f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n" Oct 30 18:30:54.382635 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:54.382620 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a11771b6-cb92-4783-bd70-4eb964d94f00-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n\" (UID: \"a11771b6-cb92-4783-bd70-4eb964d94f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n" Oct 30 18:30:54.484659 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:54.484618 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a11771b6-cb92-4783-bd70-4eb964d94f00-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n\" (UID: \"a11771b6-cb92-4783-bd70-4eb964d94f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n" Oct 30 18:30:54.484870 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:54.484672 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a11771b6-cb92-4783-bd70-4eb964d94f00-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n\" (UID: \"a11771b6-cb92-4783-bd70-4eb964d94f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n" Oct 30 18:30:54.484934 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:54.484881 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a11771b6-cb92-4783-bd70-4eb964d94f00-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n\" (UID: \"a11771b6-cb92-4783-bd70-4eb964d94f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n" Oct 30 18:30:54.484934 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:54.484916 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p77tf\" (UniqueName: \"kubernetes.io/projected/a11771b6-cb92-4783-bd70-4eb964d94f00-kube-api-access-p77tf\") pod \"maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n\" (UID: \"a11771b6-cb92-4783-bd70-4eb964d94f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n" Oct 30 18:30:54.485708 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:54.485674 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a11771b6-cb92-4783-bd70-4eb964d94f00-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n\" (UID: \"a11771b6-cb92-4783-bd70-4eb964d94f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n" Oct 30 18:30:54.487541 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:54.487514 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a11771b6-cb92-4783-bd70-4eb964d94f00-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n\" (UID: \"a11771b6-cb92-4783-bd70-4eb964d94f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n" Oct 30 18:30:54.487699 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:54.487575 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a11771b6-cb92-4783-bd70-4eb964d94f00-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n\" (UID: \"a11771b6-cb92-4783-bd70-4eb964d94f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n" Oct 30 18:30:54.510253 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:54.510221 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p77tf\" (UniqueName: \"kubernetes.io/projected/a11771b6-cb92-4783-bd70-4eb964d94f00-kube-api-access-p77tf\") pod \"maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n\" (UID: \"a11771b6-cb92-4783-bd70-4eb964d94f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n" Oct 30 18:30:54.622771 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:54.622677 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n" Oct 30 18:30:54.996865 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:30:54.996822 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda11771b6_cb92_4783_bd70_4eb964d94f00.slice/crio-0f9a4165f60665f39837aa72c376aec3dfe9a95077f5728b738c667756458269 WatchSource:0}: Error finding container 0f9a4165f60665f39837aa72c376aec3dfe9a95077f5728b738c667756458269: Status 404 returned error can't find the container with id 0f9a4165f60665f39837aa72c376aec3dfe9a95077f5728b738c667756458269 Oct 30 18:30:54.997352 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:54.997333 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n"] Oct 30 18:30:55.659998 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:55.659957 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n" event={"ID":"a11771b6-cb92-4783-bd70-4eb964d94f00","Type":"ContainerStarted","Data":"0f9a4165f60665f39837aa72c376aec3dfe9a95077f5728b738c667756458269"} Oct 30 18:30:56.670908 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:56.670868 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n" event={"ID":"a11771b6-cb92-4783-bd70-4eb964d94f00","Type":"ContainerStarted","Data":"da55ebe22dd5589f70141c2b75d52cf3090a6b3b4a969324a1a3067c471199c9"} Oct 30 18:30:56.671558 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:56.670942 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n" Oct 30 18:30:56.723904 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:30:56.723853 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n" podStartSLOduration=2.085575902 podStartE2EDuration="2.72383875s" totalImagesPullingTime="638.262848ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:56.722631062 +0000 UTC m=+2309.210268097" watchObservedRunningTime="2026-10-30 18:30:56.72383875 +0000 UTC m=+2309.211475783" Oct 30 18:31:00.387549 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:31:00.387504 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:31:02.686503 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:02.686471 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n" Oct 30 18:31:05.609201 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:05.609155 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266"] Oct 30 18:31:05.617738 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:05.617711 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266" Oct 30 18:31:05.642381 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:05.642348 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266"] Oct 30 18:31:05.650250 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:05.650216 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-3b8bf3-serving-cert\"" Oct 30 18:31:05.713638 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:05.713596 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4936cb40-0694-428e-bfd1-02a96a72a8a8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266\" (UID: \"4936cb40-0694-428e-bfd1-02a96a72a8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266" Oct 30 18:31:05.713947 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:05.713915 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4936cb40-0694-428e-bfd1-02a96a72a8a8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266\" (UID: \"4936cb40-0694-428e-bfd1-02a96a72a8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266" Oct 30 18:31:05.714187 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:05.714163 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4936cb40-0694-428e-bfd1-02a96a72a8a8-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266\" (UID: \"4936cb40-0694-428e-bfd1-02a96a72a8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266" Oct 30 18:31:05.714409 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:05.714388 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrgv7\" (UniqueName: \"kubernetes.io/projected/4936cb40-0694-428e-bfd1-02a96a72a8a8-kube-api-access-qrgv7\") pod \"maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266\" (UID: \"4936cb40-0694-428e-bfd1-02a96a72a8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266" Oct 30 18:31:05.834459 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:05.834417 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4936cb40-0694-428e-bfd1-02a96a72a8a8-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266\" (UID: \"4936cb40-0694-428e-bfd1-02a96a72a8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266" Oct 30 18:31:05.835064 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:05.835033 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrgv7\" (UniqueName: \"kubernetes.io/projected/4936cb40-0694-428e-bfd1-02a96a72a8a8-kube-api-access-qrgv7\") pod \"maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266\" (UID: \"4936cb40-0694-428e-bfd1-02a96a72a8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266" Oct 30 18:31:05.835536 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:05.835510 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4936cb40-0694-428e-bfd1-02a96a72a8a8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266\" (UID: \"4936cb40-0694-428e-bfd1-02a96a72a8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266" Oct 30 18:31:05.835769 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:05.835751 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4936cb40-0694-428e-bfd1-02a96a72a8a8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266\" (UID: \"4936cb40-0694-428e-bfd1-02a96a72a8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266" Oct 30 18:31:05.836874 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:05.836853 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4936cb40-0694-428e-bfd1-02a96a72a8a8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266\" (UID: \"4936cb40-0694-428e-bfd1-02a96a72a8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266" Oct 30 18:31:05.913406 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:05.879166 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4936cb40-0694-428e-bfd1-02a96a72a8a8-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266\" (UID: \"4936cb40-0694-428e-bfd1-02a96a72a8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266" Oct 30 18:31:05.913406 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:05.883530 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrgv7\" (UniqueName: \"kubernetes.io/projected/4936cb40-0694-428e-bfd1-02a96a72a8a8-kube-api-access-qrgv7\") pod \"maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266\" (UID: \"4936cb40-0694-428e-bfd1-02a96a72a8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266" Oct 30 18:31:05.913406 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:05.897844 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4936cb40-0694-428e-bfd1-02a96a72a8a8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266\" (UID: \"4936cb40-0694-428e-bfd1-02a96a72a8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266" Oct 30 18:31:05.936158 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:05.936110 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266" Oct 30 18:31:06.297578 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:06.297547 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266"] Oct 30 18:31:06.298670 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:31:06.298626 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4936cb40_0694_428e_bfd1_02a96a72a8a8.slice/crio-c5babd4fa55ffebde173387fb6c2122394d45c6d8d8fcdcf5700a6a81d570e43 WatchSource:0}: Error finding container c5babd4fa55ffebde173387fb6c2122394d45c6d8d8fcdcf5700a6a81d570e43: Status 404 returned error can't find the container with id c5babd4fa55ffebde173387fb6c2122394d45c6d8d8fcdcf5700a6a81d570e43 Oct 30 18:31:06.989671 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:06.989633 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266" event={"ID":"4936cb40-0694-428e-bfd1-02a96a72a8a8","Type":"ContainerStarted","Data":"c5babd4fa55ffebde173387fb6c2122394d45c6d8d8fcdcf5700a6a81d570e43"} Oct 30 18:31:07.999640 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:07.999601 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266" event={"ID":"4936cb40-0694-428e-bfd1-02a96a72a8a8","Type":"ContainerStarted","Data":"c720f5b81f703e660fbc315b474d5e3452ca5e9b9d63ce5864d993b6cbc8a60a"} Oct 30 18:31:08.000135 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:07.999670 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266" Oct 30 18:31:08.049854 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:08.049797 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266" podStartSLOduration=2.592659063 podStartE2EDuration="3.04978137s" totalImagesPullingTime="457.122307ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:31:08.045945231 +0000 UTC m=+2320.533582273" watchObservedRunningTime="2026-10-30 18:31:08.04978137 +0000 UTC m=+2320.537418452" Oct 30 18:31:14.014445 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:14.014411 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266" Oct 30 18:31:30.386258 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:31:30.386211 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:31:30.866666 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:30.866624 4455 scope.go:122] "RemoveContainer" containerID="a9f84fa6101c8e795a8d446b5021ccb549fd9517952f2308bbba9fe16fb641e3" Oct 30 18:31:41.790347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:41.789458 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" podUID="25474977-dd58-4ead-9910-f20df0701ae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.184:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:42.810348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:42.809796 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:31:43.893348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:43.869725 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" podUID="25474977-dd58-4ead-9910-f20df0701ae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.184:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:44.117532 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:43.883474 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:44.219537 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:44.219504 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:31:44.659462 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:44.630745 4455 trace.go:236] Trace[364196373]: "Calculate volume metrics of manifests for pod opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-44nvn" (30-Oct-2026 18:31:43.625) (total time: 1005ms): Oct 30 18:31:44.659462 ip-10-0-130-84 kubenswrapper[4455]: Trace[364196373]: [1.005174741s] [1.005174741s] END Oct 30 18:31:45.119211 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:45.119156 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" podUID="25474977-dd58-4ead-9910-f20df0701ae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.184:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:45.123119 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:45.122748 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" Oct 30 18:31:45.166488 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:45.166446 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:45.472864 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:45.472831 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:31:46.138034 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:46.137989 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" podUID="25474977-dd58-4ead-9910-f20df0701ae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.184:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:46.233929 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:46.232716 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:31:46.234483 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:46.234360 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" podUID="1d3471dc-aaf3-4060-a64d-2413e522e65a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:47.186350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:47.172427 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" podUID="25474977-dd58-4ead-9910-f20df0701ae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.184:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:47.208468 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:47.208417 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:47.366353 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:47.361857 4455 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:48.189981 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:48.189934 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" podUID="25474977-dd58-4ead-9910-f20df0701ae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.184:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:48.220361 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:48.219416 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:48.220361 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:48.219897 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:31:48.304347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:48.304010 4455 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" podUID="25474977-dd58-4ead-9910-f20df0701ae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.184:8000/health\": context deadline exceeded" Oct 30 18:31:49.196344 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:49.195885 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" podUID="25474977-dd58-4ead-9910-f20df0701ae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.184:8000/ready\": context deadline exceeded" Oct 30 18:31:49.226360 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:49.223614 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:50.228375 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:50.214414 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" podUID="25474977-dd58-4ead-9910-f20df0701ae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.184:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:50.237583 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:50.237541 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:50.259983 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:50.259952 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:31:50.663410 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:50.659660 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" Oct 30 18:31:56.397020 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:56.393458 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-9l4x8_909b1ff6-6d4a-4887-9b59-efe2806ac0e8/payload-processing/0.log" Oct 30 18:31:56.617055 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:31:56.617024 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-9l4x8_909b1ff6-6d4a-4887-9b59-efe2806ac0e8/payload-processing/0.log" Oct 30 18:32:00.387337 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:32:00.387281 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:32:30.097065 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:30.097023 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2"] Oct 30 18:32:30.098122 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:30.098085 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" podUID="c0d316f7-79bb-4e92-be1e-306552cc08b6" containerName="istio-proxy" containerID="cri-o://fdc0fa3b3ae4701d0e2de2b8d5636b1943a7d33c331ce8a2491303e03eefb3a1" gracePeriod=30 Oct 30 18:32:30.388754 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:32:30.388658 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:32:35.458271 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.458240 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 18:32:35.574905 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.574875 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0d316f7-79bb-4e92-be1e-306552cc08b6-kube-api-access-lcq94\" (UniqueName: \"kubernetes.io/projected/c0d316f7-79bb-4e92-be1e-306552cc08b6-kube-api-access-lcq94\") pod \"c0d316f7-79bb-4e92-be1e-306552cc08b6\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " Oct 30 18:32:35.575134 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.574985 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0d316f7-79bb-4e92-be1e-306552cc08b6-istio-token\" (UniqueName: \"kubernetes.io/projected/c0d316f7-79bb-4e92-be1e-306552cc08b6-istio-token\") pod \"c0d316f7-79bb-4e92-be1e-306552cc08b6\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " Oct 30 18:32:35.575254 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.575137 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0d316f7-79bb-4e92-be1e-306552cc08b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d316f7-79bb-4e92-be1e-306552cc08b6-istio-podinfo\") pod \"c0d316f7-79bb-4e92-be1e-306552cc08b6\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " Oct 30 18:32:35.575254 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.575233 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d316f7-79bb-4e92-be1e-306552cc08b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d316f7-79bb-4e92-be1e-306552cc08b6-credential-socket\") pod \"c0d316f7-79bb-4e92-be1e-306552cc08b6\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " Oct 30 18:32:35.575451 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.575354 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0d316f7-79bb-4e92-be1e-306552cc08b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d316f7-79bb-4e92-be1e-306552cc08b6-istiod-ca-cert\") pod \"c0d316f7-79bb-4e92-be1e-306552cc08b6\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " Oct 30 18:32:35.575513 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.575471 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d316f7-79bb-4e92-be1e-306552cc08b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d316f7-79bb-4e92-be1e-306552cc08b6-istio-envoy\") pod \"c0d316f7-79bb-4e92-be1e-306552cc08b6\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " Oct 30 18:32:35.575617 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.575563 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d316f7-79bb-4e92-be1e-306552cc08b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d316f7-79bb-4e92-be1e-306552cc08b6-workload-socket\") pod \"c0d316f7-79bb-4e92-be1e-306552cc08b6\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " Oct 30 18:32:35.575683 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.575664 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d316f7-79bb-4e92-be1e-306552cc08b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d316f7-79bb-4e92-be1e-306552cc08b6-istio-data\") pod \"c0d316f7-79bb-4e92-be1e-306552cc08b6\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " Oct 30 18:32:35.575742 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.575713 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d316f7-79bb-4e92-be1e-306552cc08b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d316f7-79bb-4e92-be1e-306552cc08b6-workload-certs\") pod \"c0d316f7-79bb-4e92-be1e-306552cc08b6\" (UID: \"c0d316f7-79bb-4e92-be1e-306552cc08b6\") " Oct 30 18:32:35.577275 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.577244 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d316f7-79bb-4e92-be1e-306552cc08b6-workload-certs" pod "c0d316f7-79bb-4e92-be1e-306552cc08b6" (UID: "c0d316f7-79bb-4e92-be1e-306552cc08b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:35.578735 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.578709 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0d316f7-79bb-4e92-be1e-306552cc08b6-istiod-ca-cert" pod "c0d316f7-79bb-4e92-be1e-306552cc08b6" (UID: "c0d316f7-79bb-4e92-be1e-306552cc08b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:32:35.581352 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.581237 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d316f7-79bb-4e92-be1e-306552cc08b6-kube-api-access-lcq94" pod "c0d316f7-79bb-4e92-be1e-306552cc08b6" (UID: "c0d316f7-79bb-4e92-be1e-306552cc08b6"). InnerVolumeSpecName "kube-api-access-lcq94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:35.581352 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.581276 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0d316f7-79bb-4e92-be1e-306552cc08b6-istio-podinfo" pod "c0d316f7-79bb-4e92-be1e-306552cc08b6" (UID: "c0d316f7-79bb-4e92-be1e-306552cc08b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:32:35.581559 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.581524 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d316f7-79bb-4e92-be1e-306552cc08b6-credential-socket" pod "c0d316f7-79bb-4e92-be1e-306552cc08b6" (UID: "c0d316f7-79bb-4e92-be1e-306552cc08b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:35.581837 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.581798 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d316f7-79bb-4e92-be1e-306552cc08b6-workload-socket" pod "c0d316f7-79bb-4e92-be1e-306552cc08b6" (UID: "c0d316f7-79bb-4e92-be1e-306552cc08b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:35.581962 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.581945 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d316f7-79bb-4e92-be1e-306552cc08b6-istio-data" pod "c0d316f7-79bb-4e92-be1e-306552cc08b6" (UID: "c0d316f7-79bb-4e92-be1e-306552cc08b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:35.584444 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.584418 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d316f7-79bb-4e92-be1e-306552cc08b6-istio-token" pod "c0d316f7-79bb-4e92-be1e-306552cc08b6" (UID: "c0d316f7-79bb-4e92-be1e-306552cc08b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:35.584552 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.584439 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d316f7-79bb-4e92-be1e-306552cc08b6-istio-envoy" pod "c0d316f7-79bb-4e92-be1e-306552cc08b6" (UID: "c0d316f7-79bb-4e92-be1e-306552cc08b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:35.678529 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.678456 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcq94\" (UniqueName: \"kubernetes.io/projected/c0d316f7-79bb-4e92-be1e-306552cc08b6-kube-api-access-lcq94\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:32:35.678529 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.678484 4455 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d316f7-79bb-4e92-be1e-306552cc08b6-istio-token\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:32:35.678529 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.678493 4455 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d316f7-79bb-4e92-be1e-306552cc08b6-istio-podinfo\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:32:35.678529 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.678501 4455 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d316f7-79bb-4e92-be1e-306552cc08b6-credential-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:32:35.678529 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.678510 4455 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d316f7-79bb-4e92-be1e-306552cc08b6-istiod-ca-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:32:35.678529 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.678517 4455 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d316f7-79bb-4e92-be1e-306552cc08b6-istio-envoy\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:32:35.678529 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.678525 4455 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d316f7-79bb-4e92-be1e-306552cc08b6-workload-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:32:35.678529 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.678532 4455 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d316f7-79bb-4e92-be1e-306552cc08b6-istio-data\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:32:35.678529 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.678543 4455 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d316f7-79bb-4e92-be1e-306552cc08b6-workload-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:32:35.799285 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.799247 4455 generic.go:356] "Generic (PLEG): container finished" podID="c0d316f7-79bb-4e92-be1e-306552cc08b6" containerID="fdc0fa3b3ae4701d0e2de2b8d5636b1943a7d33c331ce8a2491303e03eefb3a1" exitCode=0 Oct 30 18:32:35.799486 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.799295 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" Oct 30 18:32:35.799486 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.799351 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" event={"ID":"c0d316f7-79bb-4e92-be1e-306552cc08b6","Type":"ContainerDied","Data":"fdc0fa3b3ae4701d0e2de2b8d5636b1943a7d33c331ce8a2491303e03eefb3a1"} Oct 30 18:32:35.799486 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.799423 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2" event={"ID":"c0d316f7-79bb-4e92-be1e-306552cc08b6","Type":"ContainerDied","Data":"61cdefb0c85be7bc51237fad896e73094c0156dc9a66fee1ed25740e7ba168fe"} Oct 30 18:32:35.799486 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.799449 4455 scope.go:122] "RemoveContainer" containerID="fdc0fa3b3ae4701d0e2de2b8d5636b1943a7d33c331ce8a2491303e03eefb3a1" Oct 30 18:32:35.811167 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.811148 4455 scope.go:122] "RemoveContainer" containerID="8e3cc89db8afac75210239ce9725f99727895c2d2d2eaae4ccd1c57fad5b4294" Oct 30 18:32:35.822590 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.822569 4455 scope.go:122] "RemoveContainer" containerID="fdc0fa3b3ae4701d0e2de2b8d5636b1943a7d33c331ce8a2491303e03eefb3a1" Oct 30 18:32:35.822886 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:32:35.822861 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdc0fa3b3ae4701d0e2de2b8d5636b1943a7d33c331ce8a2491303e03eefb3a1\": container with ID starting with fdc0fa3b3ae4701d0e2de2b8d5636b1943a7d33c331ce8a2491303e03eefb3a1 not found: ID does not exist" containerID="fdc0fa3b3ae4701d0e2de2b8d5636b1943a7d33c331ce8a2491303e03eefb3a1" Oct 30 18:32:35.822981 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.822893 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdc0fa3b3ae4701d0e2de2b8d5636b1943a7d33c331ce8a2491303e03eefb3a1"} err="failed to get container status \"fdc0fa3b3ae4701d0e2de2b8d5636b1943a7d33c331ce8a2491303e03eefb3a1\": rpc error: code = NotFound desc = could not find container \"fdc0fa3b3ae4701d0e2de2b8d5636b1943a7d33c331ce8a2491303e03eefb3a1\": container with ID starting with fdc0fa3b3ae4701d0e2de2b8d5636b1943a7d33c331ce8a2491303e03eefb3a1 not found: ID does not exist" Oct 30 18:32:35.822981 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.822911 4455 scope.go:122] "RemoveContainer" containerID="8e3cc89db8afac75210239ce9725f99727895c2d2d2eaae4ccd1c57fad5b4294" Oct 30 18:32:35.823216 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:32:35.823195 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e3cc89db8afac75210239ce9725f99727895c2d2d2eaae4ccd1c57fad5b4294\": container with ID starting with 8e3cc89db8afac75210239ce9725f99727895c2d2d2eaae4ccd1c57fad5b4294 not found: ID does not exist" containerID="8e3cc89db8afac75210239ce9725f99727895c2d2d2eaae4ccd1c57fad5b4294" Oct 30 18:32:35.823280 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.823221 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e3cc89db8afac75210239ce9725f99727895c2d2d2eaae4ccd1c57fad5b4294"} err="failed to get container status \"8e3cc89db8afac75210239ce9725f99727895c2d2d2eaae4ccd1c57fad5b4294\": rpc error: code = NotFound desc = could not find container \"8e3cc89db8afac75210239ce9725f99727895c2d2d2eaae4ccd1c57fad5b4294\": container with ID starting with 8e3cc89db8afac75210239ce9725f99727895c2d2d2eaae4ccd1c57fad5b4294 not found: ID does not exist" Oct 30 18:32:35.860881 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.860838 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2"] Oct 30 18:32:35.865965 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:35.865932 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-h2gf2"] Oct 30 18:32:37.637511 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:37.637481 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0d316f7-79bb-4e92-be1e-306552cc08b6" path="/var/lib/kubelet/pods/c0d316f7-79bb-4e92-be1e-306552cc08b6/volumes" Oct 30 18:32:39.892808 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:39.892775 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq"] Oct 30 18:32:39.893886 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:39.893860 4455 cpu_manager.go:414] "containerMap: removing container" podUID="c0d316f7-79bb-4e92-be1e-306552cc08b6" containerName="istio-proxy" Oct 30 18:32:39.893886 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:39.893886 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0d316f7-79bb-4e92-be1e-306552cc08b6" containerName="istio-proxy" Oct 30 18:32:39.894085 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:39.893906 4455 cpu_manager.go:414] "containerMap: removing container" podUID="c0d316f7-79bb-4e92-be1e-306552cc08b6" containerName="istio-proxy" Oct 30 18:32:39.894085 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:39.893913 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0d316f7-79bb-4e92-be1e-306552cc08b6" containerName="istio-proxy" Oct 30 18:32:39.894186 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:39.894115 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0d316f7-79bb-4e92-be1e-306552cc08b6" containerName="istio-proxy" Oct 30 18:32:39.894186 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:39.894133 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0d316f7-79bb-4e92-be1e-306552cc08b6" containerName="istio-proxy" Oct 30 18:32:39.901046 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:39.901021 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" Oct 30 18:32:39.920144 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:39.920111 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq"] Oct 30 18:32:39.927707 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:39.927674 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-3b8bf3\"/\"kube-root-ca.crt\"" Oct 30 18:32:39.927883 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:39.927757 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-3b8bf3\"/\"openshift-service-ca.crt\"" Oct 30 18:32:39.927958 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:39.927899 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-3b8bf3\"/\"default-dockercfg-96fz7\"" Oct 30 18:32:39.934231 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:39.934189 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-3b8bf3\"/\"auth-test-model-3b8bf3-kserve-self-signed-certs\"" Oct 30 18:32:40.058604 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:40.058575 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38cd4356-6595-419d-a6f2-251106d67bc6-home\") pod \"auth-test-model-3b8bf3-kserve-74447bd545-lkxtq\" (UID: \"38cd4356-6595-419d-a6f2-251106d67bc6\") " pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" Oct 30 18:32:40.058815 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:40.058776 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38cd4356-6595-419d-a6f2-251106d67bc6-dshm\") pod \"auth-test-model-3b8bf3-kserve-74447bd545-lkxtq\" (UID: \"38cd4356-6595-419d-a6f2-251106d67bc6\") " pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" Oct 30 18:32:40.058913 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:40.058813 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38cd4356-6595-419d-a6f2-251106d67bc6-tmp-dir\") pod \"auth-test-model-3b8bf3-kserve-74447bd545-lkxtq\" (UID: \"38cd4356-6595-419d-a6f2-251106d67bc6\") " pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" Oct 30 18:32:40.058913 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:40.058898 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38cd4356-6595-419d-a6f2-251106d67bc6-tls-certs\") pod \"auth-test-model-3b8bf3-kserve-74447bd545-lkxtq\" (UID: \"38cd4356-6595-419d-a6f2-251106d67bc6\") " pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" Oct 30 18:32:40.059049 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:40.059028 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38cd4356-6595-419d-a6f2-251106d67bc6-model-cache\") pod \"auth-test-model-3b8bf3-kserve-74447bd545-lkxtq\" (UID: \"38cd4356-6595-419d-a6f2-251106d67bc6\") " pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" Oct 30 18:32:40.059113 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:40.059059 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqklb\" (UniqueName: \"kubernetes.io/projected/38cd4356-6595-419d-a6f2-251106d67bc6-kube-api-access-zqklb\") pod \"auth-test-model-3b8bf3-kserve-74447bd545-lkxtq\" (UID: \"38cd4356-6595-419d-a6f2-251106d67bc6\") " pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" Oct 30 18:32:40.160181 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:40.160099 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38cd4356-6595-419d-a6f2-251106d67bc6-home\") pod \"auth-test-model-3b8bf3-kserve-74447bd545-lkxtq\" (UID: \"38cd4356-6595-419d-a6f2-251106d67bc6\") " pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" Oct 30 18:32:40.160385 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:40.160223 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38cd4356-6595-419d-a6f2-251106d67bc6-dshm\") pod \"auth-test-model-3b8bf3-kserve-74447bd545-lkxtq\" (UID: \"38cd4356-6595-419d-a6f2-251106d67bc6\") " pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" Oct 30 18:32:40.160385 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:40.160243 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38cd4356-6595-419d-a6f2-251106d67bc6-tmp-dir\") pod \"auth-test-model-3b8bf3-kserve-74447bd545-lkxtq\" (UID: \"38cd4356-6595-419d-a6f2-251106d67bc6\") " pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" Oct 30 18:32:40.160385 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:40.160269 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38cd4356-6595-419d-a6f2-251106d67bc6-tls-certs\") pod \"auth-test-model-3b8bf3-kserve-74447bd545-lkxtq\" (UID: \"38cd4356-6595-419d-a6f2-251106d67bc6\") " pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" Oct 30 18:32:40.160385 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:40.160385 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38cd4356-6595-419d-a6f2-251106d67bc6-model-cache\") pod \"auth-test-model-3b8bf3-kserve-74447bd545-lkxtq\" (UID: \"38cd4356-6595-419d-a6f2-251106d67bc6\") " pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" Oct 30 18:32:40.160646 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:40.160407 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqklb\" (UniqueName: \"kubernetes.io/projected/38cd4356-6595-419d-a6f2-251106d67bc6-kube-api-access-zqklb\") pod \"auth-test-model-3b8bf3-kserve-74447bd545-lkxtq\" (UID: \"38cd4356-6595-419d-a6f2-251106d67bc6\") " pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" Oct 30 18:32:40.160646 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:40.160555 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38cd4356-6595-419d-a6f2-251106d67bc6-home\") pod \"auth-test-model-3b8bf3-kserve-74447bd545-lkxtq\" (UID: \"38cd4356-6595-419d-a6f2-251106d67bc6\") " pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" Oct 30 18:32:40.160787 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:40.160652 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38cd4356-6595-419d-a6f2-251106d67bc6-tmp-dir\") pod \"auth-test-model-3b8bf3-kserve-74447bd545-lkxtq\" (UID: \"38cd4356-6595-419d-a6f2-251106d67bc6\") " pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" Oct 30 18:32:40.160787 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:40.160716 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38cd4356-6595-419d-a6f2-251106d67bc6-model-cache\") pod \"auth-test-model-3b8bf3-kserve-74447bd545-lkxtq\" (UID: \"38cd4356-6595-419d-a6f2-251106d67bc6\") " pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" Oct 30 18:32:40.162671 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:40.162650 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38cd4356-6595-419d-a6f2-251106d67bc6-dshm\") pod \"auth-test-model-3b8bf3-kserve-74447bd545-lkxtq\" (UID: \"38cd4356-6595-419d-a6f2-251106d67bc6\") " pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" Oct 30 18:32:40.163211 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:40.163184 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38cd4356-6595-419d-a6f2-251106d67bc6-tls-certs\") pod \"auth-test-model-3b8bf3-kserve-74447bd545-lkxtq\" (UID: \"38cd4356-6595-419d-a6f2-251106d67bc6\") " pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" Oct 30 18:32:40.187000 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:40.186963 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqklb\" (UniqueName: \"kubernetes.io/projected/38cd4356-6595-419d-a6f2-251106d67bc6-kube-api-access-zqklb\") pod \"auth-test-model-3b8bf3-kserve-74447bd545-lkxtq\" (UID: \"38cd4356-6595-419d-a6f2-251106d67bc6\") " pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" Oct 30 18:32:40.221150 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:40.220695 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" Oct 30 18:32:40.741135 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:40.741045 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq"] Oct 30 18:32:40.764241 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:32:40.764216 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38cd4356_6595_419d_a6f2_251106d67bc6.slice/crio-a2aaba0cebb90e526c8a938c6b1019f8500c6d8014691f73786bcb6aeef32d9a WatchSource:0}: Error finding container a2aaba0cebb90e526c8a938c6b1019f8500c6d8014691f73786bcb6aeef32d9a: Status 404 returned error can't find the container with id a2aaba0cebb90e526c8a938c6b1019f8500c6d8014691f73786bcb6aeef32d9a Oct 30 18:32:40.844055 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:40.844023 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" event={"ID":"38cd4356-6595-419d-a6f2-251106d67bc6","Type":"ContainerStarted","Data":"a2aaba0cebb90e526c8a938c6b1019f8500c6d8014691f73786bcb6aeef32d9a"} Oct 30 18:32:41.854158 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:41.854120 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" event={"ID":"38cd4356-6595-419d-a6f2-251106d67bc6","Type":"ContainerStarted","Data":"7797cd49a441687ed491ee5b4a78b93fdac09dc81754b09dcc54c6b3c0da1ac7"} Oct 30 18:32:41.905438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:41.905384 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" podStartSLOduration=2.905364538 podStartE2EDuration="2.905364538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:41.900860944 +0000 UTC m=+2414.388497977" watchObservedRunningTime="2026-10-30 18:32:41.905364538 +0000 UTC m=+2414.393001572" Oct 30 18:32:42.221802 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:42.221770 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" Oct 30 18:32:50.221116 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:50.221075 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" Oct 30 18:32:50.235231 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:50.235207 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" Oct 30 18:32:50.944777 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:50.944747 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" Oct 30 18:32:53.017960 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:53.017920 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7"] Oct 30 18:32:53.018462 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:53.018168 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" podUID="5e30a2f8-aa4d-4dea-8468-aa09e6100c58" containerName="maas-api" containerID="cri-o://924369a89d9e84b6208d65e91061263346f386a85b508e3b633bfc1ad71f4afc" gracePeriod=30 Oct 30 18:32:54.401844 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:54.401809 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" Oct 30 18:32:54.451162 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:54.451126 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e30a2f8-aa4d-4dea-8468-aa09e6100c58-kube-api-access-cz2b9\" (UniqueName: \"kubernetes.io/projected/5e30a2f8-aa4d-4dea-8468-aa09e6100c58-kube-api-access-cz2b9\") pod \"5e30a2f8-aa4d-4dea-8468-aa09e6100c58\" (UID: \"5e30a2f8-aa4d-4dea-8468-aa09e6100c58\") " Oct 30 18:32:54.451394 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:54.451259 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e30a2f8-aa4d-4dea-8468-aa09e6100c58-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e30a2f8-aa4d-4dea-8468-aa09e6100c58-openshift-service-ca\") pod \"5e30a2f8-aa4d-4dea-8468-aa09e6100c58\" (UID: \"5e30a2f8-aa4d-4dea-8468-aa09e6100c58\") " Oct 30 18:32:54.451394 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:54.451380 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e30a2f8-aa4d-4dea-8468-aa09e6100c58-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e30a2f8-aa4d-4dea-8468-aa09e6100c58-maas-api-tls\") pod \"5e30a2f8-aa4d-4dea-8468-aa09e6100c58\" (UID: \"5e30a2f8-aa4d-4dea-8468-aa09e6100c58\") " Oct 30 18:32:54.451683 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:54.451485 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e30a2f8-aa4d-4dea-8468-aa09e6100c58-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e30a2f8-aa4d-4dea-8468-aa09e6100c58-metrics-certs\") pod \"5e30a2f8-aa4d-4dea-8468-aa09e6100c58\" (UID: \"5e30a2f8-aa4d-4dea-8468-aa09e6100c58\") " Oct 30 18:32:54.451765 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:54.451678 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e30a2f8-aa4d-4dea-8468-aa09e6100c58-openshift-service-ca" pod "5e30a2f8-aa4d-4dea-8468-aa09e6100c58" (UID: "5e30a2f8-aa4d-4dea-8468-aa09e6100c58"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:32:54.452402 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:54.452379 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e30a2f8-aa4d-4dea-8468-aa09e6100c58-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:32:54.454141 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:54.454110 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e30a2f8-aa4d-4dea-8468-aa09e6100c58-kube-api-access-cz2b9" pod "5e30a2f8-aa4d-4dea-8468-aa09e6100c58" (UID: "5e30a2f8-aa4d-4dea-8468-aa09e6100c58"). InnerVolumeSpecName "kube-api-access-cz2b9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:54.454262 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:54.454165 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e30a2f8-aa4d-4dea-8468-aa09e6100c58-maas-api-tls" pod "5e30a2f8-aa4d-4dea-8468-aa09e6100c58" (UID: "5e30a2f8-aa4d-4dea-8468-aa09e6100c58"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:32:54.454262 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:54.454165 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e30a2f8-aa4d-4dea-8468-aa09e6100c58-metrics-certs" pod "5e30a2f8-aa4d-4dea-8468-aa09e6100c58" (UID: "5e30a2f8-aa4d-4dea-8468-aa09e6100c58"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:32:54.554749 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:54.554718 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz2b9\" (UniqueName: \"kubernetes.io/projected/5e30a2f8-aa4d-4dea-8468-aa09e6100c58-kube-api-access-cz2b9\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:32:54.554749 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:54.554745 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e30a2f8-aa4d-4dea-8468-aa09e6100c58-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:32:54.554749 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:54.554754 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e30a2f8-aa4d-4dea-8468-aa09e6100c58-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:32:54.981682 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:54.981642 4455 generic.go:356] "Generic (PLEG): container finished" podID="5e30a2f8-aa4d-4dea-8468-aa09e6100c58" containerID="924369a89d9e84b6208d65e91061263346f386a85b508e3b633bfc1ad71f4afc" exitCode=0 Oct 30 18:32:54.981874 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:54.981695 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" Oct 30 18:32:54.981874 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:54.981725 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" event={"ID":"5e30a2f8-aa4d-4dea-8468-aa09e6100c58","Type":"ContainerDied","Data":"924369a89d9e84b6208d65e91061263346f386a85b508e3b633bfc1ad71f4afc"} Oct 30 18:32:54.981874 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:54.981781 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7" event={"ID":"5e30a2f8-aa4d-4dea-8468-aa09e6100c58","Type":"ContainerDied","Data":"9b74d5b69511d8c1b9812020f898641123b5df8e8eb70681422ff7c2cb3bb4a3"} Oct 30 18:32:54.981874 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:54.981809 4455 scope.go:122] "RemoveContainer" containerID="924369a89d9e84b6208d65e91061263346f386a85b508e3b633bfc1ad71f4afc" Oct 30 18:32:54.994111 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:54.994092 4455 scope.go:122] "RemoveContainer" containerID="924369a89d9e84b6208d65e91061263346f386a85b508e3b633bfc1ad71f4afc" Oct 30 18:32:54.994396 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:32:54.994373 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"924369a89d9e84b6208d65e91061263346f386a85b508e3b633bfc1ad71f4afc\": container with ID starting with 924369a89d9e84b6208d65e91061263346f386a85b508e3b633bfc1ad71f4afc not found: ID does not exist" containerID="924369a89d9e84b6208d65e91061263346f386a85b508e3b633bfc1ad71f4afc" Oct 30 18:32:54.994474 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:54.994408 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"924369a89d9e84b6208d65e91061263346f386a85b508e3b633bfc1ad71f4afc"} err="failed to get container status \"924369a89d9e84b6208d65e91061263346f386a85b508e3b633bfc1ad71f4afc\": rpc error: code = NotFound desc = could not find container \"924369a89d9e84b6208d65e91061263346f386a85b508e3b633bfc1ad71f4afc\": container with ID starting with 924369a89d9e84b6208d65e91061263346f386a85b508e3b633bfc1ad71f4afc not found: ID does not exist" Oct 30 18:32:55.053438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:55.053402 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7"] Oct 30 18:32:55.066473 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:55.066437 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-77db82-767d45c89-7jsl7"] Oct 30 18:32:55.637727 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:55.637690 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e30a2f8-aa4d-4dea-8468-aa09e6100c58" path="/var/lib/kubelet/pods/5e30a2f8-aa4d-4dea-8468-aa09e6100c58/volumes" Oct 30 18:32:59.590266 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:32:59.590224 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g"] Oct 30 18:33:00.389149 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:33:00.389112 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:33:14.624961 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:14.624916 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" podUID="25474977-dd58-4ead-9910-f20df0701ae5" containerName="main" containerID="cri-o://263a90bf4b60dc8b9c7e3ffb89c53dcd87c6ab9f02e54acb332be2b62f63e530" gracePeriod=45 Oct 30 18:33:14.939949 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:14.939919 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" Oct 30 18:33:15.016838 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:15.016798 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25474977-dd58-4ead-9910-f20df0701ae5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25474977-dd58-4ead-9910-f20df0701ae5-model-cache\") pod \"25474977-dd58-4ead-9910-f20df0701ae5\" (UID: \"25474977-dd58-4ead-9910-f20df0701ae5\") " Oct 30 18:33:15.017025 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:15.016988 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25474977-dd58-4ead-9910-f20df0701ae5-kube-api-access-6pddd\" (UniqueName: \"kubernetes.io/projected/25474977-dd58-4ead-9910-f20df0701ae5-kube-api-access-6pddd\") pod \"25474977-dd58-4ead-9910-f20df0701ae5\" (UID: \"25474977-dd58-4ead-9910-f20df0701ae5\") " Oct 30 18:33:15.017100 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:15.017048 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25474977-dd58-4ead-9910-f20df0701ae5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25474977-dd58-4ead-9910-f20df0701ae5-tmp-dir\") pod \"25474977-dd58-4ead-9910-f20df0701ae5\" (UID: \"25474977-dd58-4ead-9910-f20df0701ae5\") " Oct 30 18:33:15.017100 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:15.017078 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25474977-dd58-4ead-9910-f20df0701ae5-tls-certs\" (UniqueName: \"kubernetes.io/secret/25474977-dd58-4ead-9910-f20df0701ae5-tls-certs\") pod \"25474977-dd58-4ead-9910-f20df0701ae5\" (UID: \"25474977-dd58-4ead-9910-f20df0701ae5\") " Oct 30 18:33:15.017256 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:15.017238 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25474977-dd58-4ead-9910-f20df0701ae5-home\" (UniqueName: \"kubernetes.io/empty-dir/25474977-dd58-4ead-9910-f20df0701ae5-home\") pod \"25474977-dd58-4ead-9910-f20df0701ae5\" (UID: \"25474977-dd58-4ead-9910-f20df0701ae5\") " Oct 30 18:33:15.017425 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:15.017404 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25474977-dd58-4ead-9910-f20df0701ae5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/25474977-dd58-4ead-9910-f20df0701ae5-dshm\") pod \"25474977-dd58-4ead-9910-f20df0701ae5\" (UID: \"25474977-dd58-4ead-9910-f20df0701ae5\") " Oct 30 18:33:15.018661 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:15.017813 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25474977-dd58-4ead-9910-f20df0701ae5-home" pod "25474977-dd58-4ead-9910-f20df0701ae5" (UID: "25474977-dd58-4ead-9910-f20df0701ae5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:15.018661 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:15.017960 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25474977-dd58-4ead-9910-f20df0701ae5-model-cache" pod "25474977-dd58-4ead-9910-f20df0701ae5" (UID: "25474977-dd58-4ead-9910-f20df0701ae5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:15.018661 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:15.018502 4455 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25474977-dd58-4ead-9910-f20df0701ae5-model-cache\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:33:15.018661 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:15.018524 4455 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25474977-dd58-4ead-9910-f20df0701ae5-home\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:33:15.018661 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:15.018632 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25474977-dd58-4ead-9910-f20df0701ae5-tmp-dir" pod "25474977-dd58-4ead-9910-f20df0701ae5" (UID: "25474977-dd58-4ead-9910-f20df0701ae5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:15.020225 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:15.020202 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25474977-dd58-4ead-9910-f20df0701ae5-kube-api-access-6pddd" pod "25474977-dd58-4ead-9910-f20df0701ae5" (UID: "25474977-dd58-4ead-9910-f20df0701ae5"). InnerVolumeSpecName "kube-api-access-6pddd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:15.020652 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:15.020629 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25474977-dd58-4ead-9910-f20df0701ae5-dshm" pod "25474977-dd58-4ead-9910-f20df0701ae5" (UID: "25474977-dd58-4ead-9910-f20df0701ae5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:15.020772 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:15.020752 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25474977-dd58-4ead-9910-f20df0701ae5-tls-certs" pod "25474977-dd58-4ead-9910-f20df0701ae5" (UID: "25474977-dd58-4ead-9910-f20df0701ae5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:33:15.120076 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:15.120044 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pddd\" (UniqueName: \"kubernetes.io/projected/25474977-dd58-4ead-9910-f20df0701ae5-kube-api-access-6pddd\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:33:15.120076 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:15.120075 4455 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25474977-dd58-4ead-9910-f20df0701ae5-tmp-dir\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:33:15.120298 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:15.120087 4455 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25474977-dd58-4ead-9910-f20df0701ae5-tls-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:33:15.120298 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:15.120097 4455 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25474977-dd58-4ead-9910-f20df0701ae5-dshm\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:33:15.179718 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:15.179619 4455 generic.go:356] "Generic (PLEG): container finished" podID="25474977-dd58-4ead-9910-f20df0701ae5" containerID="263a90bf4b60dc8b9c7e3ffb89c53dcd87c6ab9f02e54acb332be2b62f63e530" exitCode=0 Oct 30 18:33:15.179718 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:15.179680 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" Oct 30 18:33:15.179950 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:15.179740 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" event={"ID":"25474977-dd58-4ead-9910-f20df0701ae5","Type":"ContainerDied","Data":"263a90bf4b60dc8b9c7e3ffb89c53dcd87c6ab9f02e54acb332be2b62f63e530"} Oct 30 18:33:15.179950 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:15.179772 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g" event={"ID":"25474977-dd58-4ead-9910-f20df0701ae5","Type":"ContainerDied","Data":"4552ae36b32d6af9d02e94423db9062e622f7fe616901cfdc139a70d5993eeb9"} Oct 30 18:33:15.179950 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:15.179792 4455 scope.go:122] "RemoveContainer" containerID="263a90bf4b60dc8b9c7e3ffb89c53dcd87c6ab9f02e54acb332be2b62f63e530" Oct 30 18:33:15.194335 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:15.194300 4455 scope.go:122] "RemoveContainer" containerID="263a90bf4b60dc8b9c7e3ffb89c53dcd87c6ab9f02e54acb332be2b62f63e530" Oct 30 18:33:15.194801 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:33:15.194780 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"263a90bf4b60dc8b9c7e3ffb89c53dcd87c6ab9f02e54acb332be2b62f63e530\": container with ID starting with 263a90bf4b60dc8b9c7e3ffb89c53dcd87c6ab9f02e54acb332be2b62f63e530 not found: ID does not exist" containerID="263a90bf4b60dc8b9c7e3ffb89c53dcd87c6ab9f02e54acb332be2b62f63e530" Oct 30 18:33:15.194890 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:15.194812 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"263a90bf4b60dc8b9c7e3ffb89c53dcd87c6ab9f02e54acb332be2b62f63e530"} err="failed to get container status \"263a90bf4b60dc8b9c7e3ffb89c53dcd87c6ab9f02e54acb332be2b62f63e530\": rpc error: code = NotFound desc = could not find container \"263a90bf4b60dc8b9c7e3ffb89c53dcd87c6ab9f02e54acb332be2b62f63e530\": container with ID starting with 263a90bf4b60dc8b9c7e3ffb89c53dcd87c6ab9f02e54acb332be2b62f63e530 not found: ID does not exist" Oct 30 18:33:15.239333 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:15.239290 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g"] Oct 30 18:33:15.247490 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:15.247463 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-77db82/ipp-route-77db82-kserve-6464b5557-zxz9g"] Oct 30 18:33:15.636919 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:15.636884 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25474977-dd58-4ead-9910-f20df0701ae5" path="/var/lib/kubelet/pods/25474977-dd58-4ead-9910-f20df0701ae5/volumes" Oct 30 18:33:23.005350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:23.004439 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" podUID="38cd4356-6595-419d-a6f2-251106d67bc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.204:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:23.065491 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:23.065447 4455 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:33:23.065822 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:23.065787 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" podUID="bbf4bc50-3670-40de-8159-3d8805fdd601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:23.242459 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:23.242407 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:23.292523 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:23.292412 4455 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:33:23.292806 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:23.292775 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" podUID="3daa8589-9def-486f-82ee-0c923a18b68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:24.010939 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:24.008889 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" podUID="38cd4356-6595-419d-a6f2-251106d67bc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.204:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:24.279433 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:24.279311 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:33:24.392350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:24.321367 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" Oct 30 18:33:24.421351 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:24.396458 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:33:24.541408 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:24.533448 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" Oct 30 18:33:24.977348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:24.973714 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:33:26.128336 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:26.128271 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:26.661382 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:26.655543 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" podUID="38cd4356-6595-419d-a6f2-251106d67bc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.204:8000/ready\": context deadline exceeded" Oct 30 18:33:27.216350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:27.191411 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:33:27.248596 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:27.248564 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:33:27.662486 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:27.662375 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" podUID="38cd4356-6595-419d-a6f2-251106d67bc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.204:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:27.836380 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:27.835944 4455 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:28.141350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:28.138413 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" Oct 30 18:33:28.192641 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:28.192593 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:33:28.606721 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:28.606357 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" Oct 30 18:33:28.680351 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:28.677482 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:33:28.680351 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:28.677555 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" podUID="c7c21be6-b9eb-401c-aa3c-af5b62960cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:29.013982 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:29.013942 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:33:29.014338 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:29.014270 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" podUID="4e7b0c73-0041-4549-b47e-76a7aaed2b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:29.221349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:29.214152 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:33:29.694346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:29.680002 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" podUID="38cd4356-6595-419d-a6f2-251106d67bc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.204:8000/ready\": context deadline exceeded" Oct 30 18:33:30.577384 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:30.563160 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:33:31.041946 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:33:31.041777 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:33:41.857598 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:41.857518 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv"] Oct 30 18:33:41.858436 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:41.858418 4455 cpu_manager.go:414] "containerMap: removing container" podUID="5e30a2f8-aa4d-4dea-8468-aa09e6100c58" containerName="maas-api" Oct 30 18:33:41.858436 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:41.858436 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e30a2f8-aa4d-4dea-8468-aa09e6100c58" containerName="maas-api" Oct 30 18:33:41.858556 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:41.858461 4455 cpu_manager.go:414] "containerMap: removing container" podUID="25474977-dd58-4ead-9910-f20df0701ae5" containerName="main" Oct 30 18:33:41.858556 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:41.858466 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25474977-dd58-4ead-9910-f20df0701ae5" containerName="main" Oct 30 18:33:41.858649 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:41.858585 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e30a2f8-aa4d-4dea-8468-aa09e6100c58" containerName="maas-api" Oct 30 18:33:41.858649 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:41.858599 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="25474977-dd58-4ead-9910-f20df0701ae5" containerName="main" Oct 30 18:33:41.864100 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:41.864078 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" Oct 30 18:33:41.891492 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:41.891461 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv"] Oct 30 18:33:41.939581 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:41.939533 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b41517e2-cc07-4530-8ec9-092c2a92ad47-workload-certs\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" Oct 30 18:33:41.939746 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:41.939663 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b41517e2-cc07-4530-8ec9-092c2a92ad47-istio-envoy\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" Oct 30 18:33:41.939746 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:41.939716 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b41517e2-cc07-4530-8ec9-092c2a92ad47-istio-token\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" Oct 30 18:33:41.939880 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:41.939863 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41517e2-cc07-4530-8ec9-092c2a92ad47-credential-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" Oct 30 18:33:41.939934 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:41.939918 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqfzv\" (UniqueName: \"kubernetes.io/projected/b41517e2-cc07-4530-8ec9-092c2a92ad47-kube-api-access-nqfzv\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" Oct 30 18:33:41.940011 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:41.939988 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b41517e2-cc07-4530-8ec9-092c2a92ad47-istio-data\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" Oct 30 18:33:41.940148 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:41.940130 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b41517e2-cc07-4530-8ec9-092c2a92ad47-istio-podinfo\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" Oct 30 18:33:41.940279 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:41.940264 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b41517e2-cc07-4530-8ec9-092c2a92ad47-istiod-ca-cert\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" Oct 30 18:33:41.940403 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:41.940381 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41517e2-cc07-4530-8ec9-092c2a92ad47-workload-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" Oct 30 18:33:41.961092 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:41.961057 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-b00d69-openshift-default-dockercfg-dp9gr\"" Oct 30 18:33:42.042138 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:42.042102 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b41517e2-cc07-4530-8ec9-092c2a92ad47-workload-certs\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" Oct 30 18:33:42.042355 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:42.042156 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b41517e2-cc07-4530-8ec9-092c2a92ad47-istio-envoy\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" Oct 30 18:33:42.042355 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:42.042179 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b41517e2-cc07-4530-8ec9-092c2a92ad47-istio-token\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" Oct 30 18:33:42.042355 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:42.042253 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41517e2-cc07-4530-8ec9-092c2a92ad47-credential-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" Oct 30 18:33:42.042355 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:42.042284 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqfzv\" (UniqueName: \"kubernetes.io/projected/b41517e2-cc07-4530-8ec9-092c2a92ad47-kube-api-access-nqfzv\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" Oct 30 18:33:42.042355 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:42.042300 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b41517e2-cc07-4530-8ec9-092c2a92ad47-istio-data\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" Oct 30 18:33:42.042720 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:42.042363 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b41517e2-cc07-4530-8ec9-092c2a92ad47-istio-podinfo\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" Oct 30 18:33:42.042720 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:42.042477 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b41517e2-cc07-4530-8ec9-092c2a92ad47-istiod-ca-cert\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" Oct 30 18:33:42.042720 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:42.042576 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b41517e2-cc07-4530-8ec9-092c2a92ad47-workload-certs\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" Oct 30 18:33:42.042720 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:42.042615 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41517e2-cc07-4530-8ec9-092c2a92ad47-workload-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" Oct 30 18:33:42.042720 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:42.042701 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b41517e2-cc07-4530-8ec9-092c2a92ad47-istio-data\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" Oct 30 18:33:42.042996 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:42.042783 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41517e2-cc07-4530-8ec9-092c2a92ad47-credential-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" Oct 30 18:33:42.042996 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:42.042908 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41517e2-cc07-4530-8ec9-092c2a92ad47-workload-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" Oct 30 18:33:42.043288 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:42.043263 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b41517e2-cc07-4530-8ec9-092c2a92ad47-istiod-ca-cert\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" Oct 30 18:33:42.045075 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:42.045051 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b41517e2-cc07-4530-8ec9-092c2a92ad47-istio-podinfo\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" Oct 30 18:33:42.045187 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:42.045089 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b41517e2-cc07-4530-8ec9-092c2a92ad47-istio-envoy\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" Oct 30 18:33:42.096762 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:42.096681 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b41517e2-cc07-4530-8ec9-092c2a92ad47-istio-token\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" Oct 30 18:33:42.097516 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:42.097493 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqfzv\" (UniqueName: \"kubernetes.io/projected/b41517e2-cc07-4530-8ec9-092c2a92ad47-kube-api-access-nqfzv\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" Oct 30 18:33:42.180461 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:42.180387 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" Oct 30 18:33:42.521306 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:33:42.521274 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb41517e2_cc07_4530_8ec9_092c2a92ad47.slice/crio-1c6d6a053664ef57cae6df2c57976dc6f629cb80a4b269396431a670825a2e10 WatchSource:0}: Error finding container 1c6d6a053664ef57cae6df2c57976dc6f629cb80a4b269396431a670825a2e10: Status 404 returned error can't find the container with id 1c6d6a053664ef57cae6df2c57976dc6f629cb80a4b269396431a670825a2e10 Oct 30 18:33:42.521809 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:42.521660 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv"] Oct 30 18:33:42.524543 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:42.524513 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:33:42.524612 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:42.524576 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:33:42.524612 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:42.524604 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:33:42.643341 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:42.643288 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" event={"ID":"b41517e2-cc07-4530-8ec9-092c2a92ad47","Type":"ContainerStarted","Data":"ce80d0cadfc75aac9e1f53d048259d4126674e32cbe065b2a660e32f08699c89"} Oct 30 18:33:42.643535 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:42.643364 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" event={"ID":"b41517e2-cc07-4530-8ec9-092c2a92ad47","Type":"ContainerStarted","Data":"1c6d6a053664ef57cae6df2c57976dc6f629cb80a4b269396431a670825a2e10"} Oct 30 18:33:42.695635 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:42.695573 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" podStartSLOduration=1.695555326 podStartE2EDuration="1.695555326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:42.691829763 +0000 UTC m=+2475.179466795" watchObservedRunningTime="2026-10-30 18:33:42.695555326 +0000 UTC m=+2475.183192362" Oct 30 18:33:43.181339 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:43.181269 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" Oct 30 18:33:43.182909 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:43.182882 4455 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 30 18:33:43.183041 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:43.182934 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" podUID="b41517e2-cc07-4530-8ec9-092c2a92ad47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 30 18:33:44.188350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:44.186749 4455 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 30 18:33:44.188350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:44.186829 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" podUID="b41517e2-cc07-4530-8ec9-092c2a92ad47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 30 18:33:44.634383 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:44.634351 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9"] Oct 30 18:33:44.634835 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:44.634809 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" podUID="1d98345b-38d7-49b3-8092-a41b1f584fb0" containerName="istio-proxy" containerID="cri-o://9d5b84264e6fac78fedbd2349b91b34feda797d9563b0d8e98931812624e7b84" gracePeriod=30 Oct 30 18:33:44.648673 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:44.648634 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg"] Oct 30 18:33:44.648899 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:44.648865 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" podUID="7894da9d-a98c-45ac-9968-bfe4e9747c49" containerName="istio-proxy" containerID="cri-o://ecccd275d82caf62c5c903c6bb4bfd1a8a1712ef2ca1485658f10801bba111a2" gracePeriod=30 Oct 30 18:33:45.181104 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:45.181066 4455 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 30 18:33:45.181440 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:45.181392 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" podUID="b41517e2-cc07-4530-8ec9-092c2a92ad47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 30 18:33:46.183349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:46.181736 4455 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 30 18:33:46.183349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:46.181803 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" podUID="b41517e2-cc07-4530-8ec9-092c2a92ad47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 30 18:33:47.181772 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:47.181734 4455 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 30 18:33:47.182068 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:47.182040 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" podUID="b41517e2-cc07-4530-8ec9-092c2a92ad47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 30 18:33:48.191671 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:48.191635 4455 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 30 18:33:48.192275 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:48.192244 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" podUID="b41517e2-cc07-4530-8ec9-092c2a92ad47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 30 18:33:49.175747 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:49.175707 4455 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:33:49.175947 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:49.175775 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" podUID="1d98345b-38d7-49b3-8092-a41b1f584fb0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:33:49.181108 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:49.181078 4455 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 30 18:33:49.181272 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:49.181125 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" podUID="b41517e2-cc07-4530-8ec9-092c2a92ad47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 30 18:33:49.826416 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:49.825908 4455 generic.go:356] "Generic (PLEG): container finished" podID="1d98345b-38d7-49b3-8092-a41b1f584fb0" containerID="9d5b84264e6fac78fedbd2349b91b34feda797d9563b0d8e98931812624e7b84" exitCode=0 Oct 30 18:33:49.826416 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:49.826289 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" event={"ID":"1d98345b-38d7-49b3-8092-a41b1f584fb0","Type":"ContainerDied","Data":"9d5b84264e6fac78fedbd2349b91b34feda797d9563b0d8e98931812624e7b84"} Oct 30 18:33:49.829887 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:49.829763 4455 generic.go:356] "Generic (PLEG): container finished" podID="7894da9d-a98c-45ac-9968-bfe4e9747c49" containerID="ecccd275d82caf62c5c903c6bb4bfd1a8a1712ef2ca1485658f10801bba111a2" exitCode=0 Oct 30 18:33:49.829887 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:49.829807 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" event={"ID":"7894da9d-a98c-45ac-9968-bfe4e9747c49","Type":"ContainerDied","Data":"ecccd275d82caf62c5c903c6bb4bfd1a8a1712ef2ca1485658f10801bba111a2"} Oct 30 18:33:50.086699 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.086676 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" Oct 30 18:33:50.091990 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.091968 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" Oct 30 18:33:50.204265 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.204231 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" Oct 30 18:33:50.204854 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.204689 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" Oct 30 18:33:50.206080 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.206058 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" Oct 30 18:33:50.264118 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.264088 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d98345b-38d7-49b3-8092-a41b1f584fb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d98345b-38d7-49b3-8092-a41b1f584fb0-istio-podinfo\") pod \"1d98345b-38d7-49b3-8092-a41b1f584fb0\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " Oct 30 18:33:50.264336 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.264140 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d98345b-38d7-49b3-8092-a41b1f584fb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98345b-38d7-49b3-8092-a41b1f584fb0-workload-socket\") pod \"1d98345b-38d7-49b3-8092-a41b1f584fb0\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " Oct 30 18:33:50.264336 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.264180 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d98345b-38d7-49b3-8092-a41b1f584fb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d98345b-38d7-49b3-8092-a41b1f584fb0-workload-certs\") pod \"1d98345b-38d7-49b3-8092-a41b1f584fb0\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " Oct 30 18:33:50.264336 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.264201 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d98345b-38d7-49b3-8092-a41b1f584fb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d98345b-38d7-49b3-8092-a41b1f584fb0-istio-data\") pod \"1d98345b-38d7-49b3-8092-a41b1f584fb0\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " Oct 30 18:33:50.264336 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.264261 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d98345b-38d7-49b3-8092-a41b1f584fb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d98345b-38d7-49b3-8092-a41b1f584fb0-istio-envoy\") pod \"1d98345b-38d7-49b3-8092-a41b1f584fb0\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " Oct 30 18:33:50.264336 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.264292 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7894da9d-a98c-45ac-9968-bfe4e9747c49-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7894da9d-a98c-45ac-9968-bfe4e9747c49-istio-podinfo\") pod \"7894da9d-a98c-45ac-9968-bfe4e9747c49\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " Oct 30 18:33:50.264618 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.264406 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7894da9d-a98c-45ac-9968-bfe4e9747c49-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7894da9d-a98c-45ac-9968-bfe4e9747c49-workload-socket\") pod \"7894da9d-a98c-45ac-9968-bfe4e9747c49\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " Oct 30 18:33:50.264618 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.264438 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d98345b-38d7-49b3-8092-a41b1f584fb0-istio-token\" (UniqueName: \"kubernetes.io/projected/1d98345b-38d7-49b3-8092-a41b1f584fb0-istio-token\") pod \"1d98345b-38d7-49b3-8092-a41b1f584fb0\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " Oct 30 18:33:50.264618 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.264511 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7894da9d-a98c-45ac-9968-bfe4e9747c49-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7894da9d-a98c-45ac-9968-bfe4e9747c49-istio-data\") pod \"7894da9d-a98c-45ac-9968-bfe4e9747c49\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " Oct 30 18:33:50.264618 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.264571 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d98345b-38d7-49b3-8092-a41b1f584fb0-workload-certs" pod "1d98345b-38d7-49b3-8092-a41b1f584fb0" (UID: "1d98345b-38d7-49b3-8092-a41b1f584fb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.264752 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.264616 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7894da9d-a98c-45ac-9968-bfe4e9747c49-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7894da9d-a98c-45ac-9968-bfe4e9747c49-istiod-ca-cert\") pod \"7894da9d-a98c-45ac-9968-bfe4e9747c49\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " Oct 30 18:33:50.264752 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.264675 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7894da9d-a98c-45ac-9968-bfe4e9747c49-kube-api-access-zkscg\" (UniqueName: \"kubernetes.io/projected/7894da9d-a98c-45ac-9968-bfe4e9747c49-kube-api-access-zkscg\") pod \"7894da9d-a98c-45ac-9968-bfe4e9747c49\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " Oct 30 18:33:50.264818 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.264756 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d98345b-38d7-49b3-8092-a41b1f584fb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d98345b-38d7-49b3-8092-a41b1f584fb0-istiod-ca-cert\") pod \"1d98345b-38d7-49b3-8092-a41b1f584fb0\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " Oct 30 18:33:50.264877 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.264836 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7894da9d-a98c-45ac-9968-bfe4e9747c49-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7894da9d-a98c-45ac-9968-bfe4e9747c49-credential-socket\") pod \"7894da9d-a98c-45ac-9968-bfe4e9747c49\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " Oct 30 18:33:50.264877 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.264847 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7894da9d-a98c-45ac-9968-bfe4e9747c49-workload-socket" pod "7894da9d-a98c-45ac-9968-bfe4e9747c49" (UID: "7894da9d-a98c-45ac-9968-bfe4e9747c49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.264877 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.264866 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d98345b-38d7-49b3-8092-a41b1f584fb0-workload-socket" pod "1d98345b-38d7-49b3-8092-a41b1f584fb0" (UID: "1d98345b-38d7-49b3-8092-a41b1f584fb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.265024 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.264879 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7894da9d-a98c-45ac-9968-bfe4e9747c49-istio-token\" (UniqueName: \"kubernetes.io/projected/7894da9d-a98c-45ac-9968-bfe4e9747c49-istio-token\") pod \"7894da9d-a98c-45ac-9968-bfe4e9747c49\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " Oct 30 18:33:50.265024 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.264924 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7894da9d-a98c-45ac-9968-bfe4e9747c49-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7894da9d-a98c-45ac-9968-bfe4e9747c49-workload-certs\") pod \"7894da9d-a98c-45ac-9968-bfe4e9747c49\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " Oct 30 18:33:50.265024 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.264993 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d98345b-38d7-49b3-8092-a41b1f584fb0-kube-api-access-9tghn\" (UniqueName: \"kubernetes.io/projected/1d98345b-38d7-49b3-8092-a41b1f584fb0-kube-api-access-9tghn\") pod \"1d98345b-38d7-49b3-8092-a41b1f584fb0\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " Oct 30 18:33:50.265024 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.265001 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7894da9d-a98c-45ac-9968-bfe4e9747c49-istio-data" pod "7894da9d-a98c-45ac-9968-bfe4e9747c49" (UID: "7894da9d-a98c-45ac-9968-bfe4e9747c49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.265456 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.265033 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7894da9d-a98c-45ac-9968-bfe4e9747c49-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7894da9d-a98c-45ac-9968-bfe4e9747c49-istio-envoy\") pod \"7894da9d-a98c-45ac-9968-bfe4e9747c49\" (UID: \"7894da9d-a98c-45ac-9968-bfe4e9747c49\") " Oct 30 18:33:50.265456 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.265065 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d98345b-38d7-49b3-8092-a41b1f584fb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98345b-38d7-49b3-8092-a41b1f584fb0-credential-socket\") pod \"1d98345b-38d7-49b3-8092-a41b1f584fb0\" (UID: \"1d98345b-38d7-49b3-8092-a41b1f584fb0\") " Oct 30 18:33:50.265456 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.265222 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d98345b-38d7-49b3-8092-a41b1f584fb0-istiod-ca-cert" pod "1d98345b-38d7-49b3-8092-a41b1f584fb0" (UID: "1d98345b-38d7-49b3-8092-a41b1f584fb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:50.265456 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.265438 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d98345b-38d7-49b3-8092-a41b1f584fb0-istio-data" pod "1d98345b-38d7-49b3-8092-a41b1f584fb0" (UID: "1d98345b-38d7-49b3-8092-a41b1f584fb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.265681 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.265651 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7894da9d-a98c-45ac-9968-bfe4e9747c49-credential-socket" pod "7894da9d-a98c-45ac-9968-bfe4e9747c49" (UID: "7894da9d-a98c-45ac-9968-bfe4e9747c49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.265681 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.265581 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7894da9d-a98c-45ac-9968-bfe4e9747c49-istiod-ca-cert" pod "7894da9d-a98c-45ac-9968-bfe4e9747c49" (UID: "7894da9d-a98c-45ac-9968-bfe4e9747c49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:50.266548 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.266158 4455 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7894da9d-a98c-45ac-9968-bfe4e9747c49-credential-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.266548 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.266187 4455 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98345b-38d7-49b3-8092-a41b1f584fb0-workload-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.266548 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.266201 4455 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d98345b-38d7-49b3-8092-a41b1f584fb0-workload-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.266548 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.266215 4455 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d98345b-38d7-49b3-8092-a41b1f584fb0-istio-data\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.266548 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.266228 4455 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7894da9d-a98c-45ac-9968-bfe4e9747c49-workload-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.266548 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.266240 4455 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7894da9d-a98c-45ac-9968-bfe4e9747c49-istio-data\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.266548 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.266254 4455 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7894da9d-a98c-45ac-9968-bfe4e9747c49-istiod-ca-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.266548 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.266266 4455 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d98345b-38d7-49b3-8092-a41b1f584fb0-istiod-ca-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.267695 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.267642 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d98345b-38d7-49b3-8092-a41b1f584fb0-credential-socket" pod "1d98345b-38d7-49b3-8092-a41b1f584fb0" (UID: "1d98345b-38d7-49b3-8092-a41b1f584fb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.267955 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.267922 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7894da9d-a98c-45ac-9968-bfe4e9747c49-workload-certs" pod "7894da9d-a98c-45ac-9968-bfe4e9747c49" (UID: "7894da9d-a98c-45ac-9968-bfe4e9747c49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.270843 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.270820 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7894da9d-a98c-45ac-9968-bfe4e9747c49-kube-api-access-zkscg" pod "7894da9d-a98c-45ac-9968-bfe4e9747c49" (UID: "7894da9d-a98c-45ac-9968-bfe4e9747c49"). InnerVolumeSpecName "kube-api-access-zkscg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:50.271534 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.271511 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d98345b-38d7-49b3-8092-a41b1f584fb0-istio-envoy" pod "1d98345b-38d7-49b3-8092-a41b1f584fb0" (UID: "1d98345b-38d7-49b3-8092-a41b1f584fb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.271791 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.271730 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d98345b-38d7-49b3-8092-a41b1f584fb0-istio-podinfo" pod "1d98345b-38d7-49b3-8092-a41b1f584fb0" (UID: "1d98345b-38d7-49b3-8092-a41b1f584fb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:33:50.272511 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.272487 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d98345b-38d7-49b3-8092-a41b1f584fb0-kube-api-access-9tghn" pod "1d98345b-38d7-49b3-8092-a41b1f584fb0" (UID: "1d98345b-38d7-49b3-8092-a41b1f584fb0"). InnerVolumeSpecName "kube-api-access-9tghn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:50.272807 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.272782 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d98345b-38d7-49b3-8092-a41b1f584fb0-istio-token" pod "1d98345b-38d7-49b3-8092-a41b1f584fb0" (UID: "1d98345b-38d7-49b3-8092-a41b1f584fb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:50.273246 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.273225 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7894da9d-a98c-45ac-9968-bfe4e9747c49-istio-token" pod "7894da9d-a98c-45ac-9968-bfe4e9747c49" (UID: "7894da9d-a98c-45ac-9968-bfe4e9747c49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:50.273652 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.273627 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7894da9d-a98c-45ac-9968-bfe4e9747c49-istio-envoy" pod "7894da9d-a98c-45ac-9968-bfe4e9747c49" (UID: "7894da9d-a98c-45ac-9968-bfe4e9747c49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.273773 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.273755 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7894da9d-a98c-45ac-9968-bfe4e9747c49-istio-podinfo" pod "7894da9d-a98c-45ac-9968-bfe4e9747c49" (UID: "7894da9d-a98c-45ac-9968-bfe4e9747c49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:33:50.368776 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.368743 4455 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d98345b-38d7-49b3-8092-a41b1f584fb0-istio-podinfo\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.368972 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.368960 4455 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d98345b-38d7-49b3-8092-a41b1f584fb0-istio-envoy\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.369084 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.369074 4455 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7894da9d-a98c-45ac-9968-bfe4e9747c49-istio-podinfo\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.369236 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.369225 4455 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d98345b-38d7-49b3-8092-a41b1f584fb0-istio-token\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.369364 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.369352 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkscg\" (UniqueName: \"kubernetes.io/projected/7894da9d-a98c-45ac-9968-bfe4e9747c49-kube-api-access-zkscg\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.369451 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.369439 4455 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7894da9d-a98c-45ac-9968-bfe4e9747c49-istio-token\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.369535 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.369526 4455 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7894da9d-a98c-45ac-9968-bfe4e9747c49-workload-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.369615 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.369605 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tghn\" (UniqueName: \"kubernetes.io/projected/1d98345b-38d7-49b3-8092-a41b1f584fb0-kube-api-access-9tghn\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.369695 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.369685 4455 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7894da9d-a98c-45ac-9968-bfe4e9747c49-istio-envoy\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.369775 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.369764 4455 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98345b-38d7-49b3-8092-a41b1f584fb0-credential-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.839738 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.839707 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" Oct 30 18:33:50.839738 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.839725 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg" event={"ID":"7894da9d-a98c-45ac-9968-bfe4e9747c49","Type":"ContainerDied","Data":"2947968309e4d68198583c7aae8c870089087cef0bf82ac2cd3c7883844581db"} Oct 30 18:33:50.840383 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.839794 4455 scope.go:122] "RemoveContainer" containerID="ecccd275d82caf62c5c903c6bb4bfd1a8a1712ef2ca1485658f10801bba111a2" Oct 30 18:33:50.841420 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.841395 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" event={"ID":"1d98345b-38d7-49b3-8092-a41b1f584fb0","Type":"ContainerDied","Data":"dc0a6124b03c57f3bff71f1964ae97e56c1da23ce1622db95e5ce3431c319a8c"} Oct 30 18:33:50.841597 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.841415 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9" Oct 30 18:33:50.852667 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.852650 4455 scope.go:122] "RemoveContainer" containerID="9d5b84264e6fac78fedbd2349b91b34feda797d9563b0d8e98931812624e7b84" Oct 30 18:33:50.917702 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.917670 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9"] Oct 30 18:33:50.928562 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.928531 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-2jvc9"] Oct 30 18:33:50.958676 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.958597 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg"] Oct 30 18:33:50.962873 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:50.962847 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-6tpkg"] Oct 30 18:33:51.637238 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:51.637198 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d98345b-38d7-49b3-8092-a41b1f584fb0" path="/var/lib/kubelet/pods/1d98345b-38d7-49b3-8092-a41b1f584fb0/volumes" Oct 30 18:33:51.637819 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:33:51.637802 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7894da9d-a98c-45ac-9968-bfe4e9747c49" path="/var/lib/kubelet/pods/7894da9d-a98c-45ac-9968-bfe4e9747c49/volumes" Oct 30 18:34:00.386021 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:34:00.385983 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:34:02.261398 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:02.261359 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs"] Oct 30 18:34:02.261897 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:02.261621 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs" podUID="b0dbf354-3013-4694-b003-a55b1ddbaea0" containerName="maas-api" containerID="cri-o://687ad10b726e1e5ea64de1d35575778fc34cf04bf02a83c808c03a2447357815" gracePeriod=30 Oct 30 18:34:02.584832 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:02.584803 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs" Oct 30 18:34:02.706228 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:02.706185 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0dbf354-3013-4694-b003-a55b1ddbaea0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0dbf354-3013-4694-b003-a55b1ddbaea0-openshift-service-ca\") pod \"b0dbf354-3013-4694-b003-a55b1ddbaea0\" (UID: \"b0dbf354-3013-4694-b003-a55b1ddbaea0\") " Oct 30 18:34:02.706454 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:02.706417 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0dbf354-3013-4694-b003-a55b1ddbaea0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0dbf354-3013-4694-b003-a55b1ddbaea0-maas-api-tls\") pod \"b0dbf354-3013-4694-b003-a55b1ddbaea0\" (UID: \"b0dbf354-3013-4694-b003-a55b1ddbaea0\") " Oct 30 18:34:02.706575 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:02.706559 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0dbf354-3013-4694-b003-a55b1ddbaea0-kube-api-access-vpk88\" (UniqueName: \"kubernetes.io/projected/b0dbf354-3013-4694-b003-a55b1ddbaea0-kube-api-access-vpk88\") pod \"b0dbf354-3013-4694-b003-a55b1ddbaea0\" (UID: \"b0dbf354-3013-4694-b003-a55b1ddbaea0\") " Oct 30 18:34:02.706801 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:02.706785 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0dbf354-3013-4694-b003-a55b1ddbaea0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0dbf354-3013-4694-b003-a55b1ddbaea0-metrics-certs\") pod \"b0dbf354-3013-4694-b003-a55b1ddbaea0\" (UID: \"b0dbf354-3013-4694-b003-a55b1ddbaea0\") " Oct 30 18:34:02.708794 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:02.708748 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0dbf354-3013-4694-b003-a55b1ddbaea0-openshift-service-ca" pod "b0dbf354-3013-4694-b003-a55b1ddbaea0" (UID: "b0dbf354-3013-4694-b003-a55b1ddbaea0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:34:02.722383 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:02.721772 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0dbf354-3013-4694-b003-a55b1ddbaea0-maas-api-tls" pod "b0dbf354-3013-4694-b003-a55b1ddbaea0" (UID: "b0dbf354-3013-4694-b003-a55b1ddbaea0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:34:02.722530 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:02.722477 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0dbf354-3013-4694-b003-a55b1ddbaea0-metrics-certs" pod "b0dbf354-3013-4694-b003-a55b1ddbaea0" (UID: "b0dbf354-3013-4694-b003-a55b1ddbaea0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:34:02.723178 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:02.723157 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0dbf354-3013-4694-b003-a55b1ddbaea0-kube-api-access-vpk88" pod "b0dbf354-3013-4694-b003-a55b1ddbaea0" (UID: "b0dbf354-3013-4694-b003-a55b1ddbaea0"). InnerVolumeSpecName "kube-api-access-vpk88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:02.811399 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:02.810950 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0dbf354-3013-4694-b003-a55b1ddbaea0-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:34:02.811399 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:02.810997 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0dbf354-3013-4694-b003-a55b1ddbaea0-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:34:02.811399 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:02.811012 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0dbf354-3013-4694-b003-a55b1ddbaea0-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:34:02.811399 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:02.811026 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpk88\" (UniqueName: \"kubernetes.io/projected/b0dbf354-3013-4694-b003-a55b1ddbaea0-kube-api-access-vpk88\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:34:02.959346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:02.959169 4455 generic.go:356] "Generic (PLEG): container finished" podID="b0dbf354-3013-4694-b003-a55b1ddbaea0" containerID="687ad10b726e1e5ea64de1d35575778fc34cf04bf02a83c808c03a2447357815" exitCode=0 Oct 30 18:34:02.959346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:02.959267 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs" event={"ID":"b0dbf354-3013-4694-b003-a55b1ddbaea0","Type":"ContainerDied","Data":"687ad10b726e1e5ea64de1d35575778fc34cf04bf02a83c808c03a2447357815"} Oct 30 18:34:02.959631 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:02.959362 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs" event={"ID":"b0dbf354-3013-4694-b003-a55b1ddbaea0","Type":"ContainerDied","Data":"7560614abc7406ac6016b9940d204e394464b357f4a8dbb0cebdfc53240cf3db"} Oct 30 18:34:02.959631 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:02.959390 4455 scope.go:122] "RemoveContainer" containerID="687ad10b726e1e5ea64de1d35575778fc34cf04bf02a83c808c03a2447357815" Oct 30 18:34:02.959631 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:02.959554 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs" Oct 30 18:34:03.017565 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:03.012817 4455 scope.go:122] "RemoveContainer" containerID="687ad10b726e1e5ea64de1d35575778fc34cf04bf02a83c808c03a2447357815" Oct 30 18:34:03.017565 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:34:03.013161 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"687ad10b726e1e5ea64de1d35575778fc34cf04bf02a83c808c03a2447357815\": container with ID starting with 687ad10b726e1e5ea64de1d35575778fc34cf04bf02a83c808c03a2447357815 not found: ID does not exist" containerID="687ad10b726e1e5ea64de1d35575778fc34cf04bf02a83c808c03a2447357815" Oct 30 18:34:03.017565 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:03.013187 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"687ad10b726e1e5ea64de1d35575778fc34cf04bf02a83c808c03a2447357815"} err="failed to get container status \"687ad10b726e1e5ea64de1d35575778fc34cf04bf02a83c808c03a2447357815\": rpc error: code = NotFound desc = could not find container \"687ad10b726e1e5ea64de1d35575778fc34cf04bf02a83c808c03a2447357815\": container with ID starting with 687ad10b726e1e5ea64de1d35575778fc34cf04bf02a83c808c03a2447357815 not found: ID does not exist" Oct 30 18:34:03.057690 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:03.057041 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs"] Oct 30 18:34:03.064459 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:03.062936 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-602e18-6544d5b754-crvfs"] Oct 30 18:34:03.636335 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:03.636289 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0dbf354-3013-4694-b003-a55b1ddbaea0" path="/var/lib/kubelet/pods/b0dbf354-3013-4694-b003-a55b1ddbaea0/volumes" Oct 30 18:34:26.975456 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:26.975418 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6"] Oct 30 18:34:26.976429 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:26.976411 4455 cpu_manager.go:414] "containerMap: removing container" podUID="1d98345b-38d7-49b3-8092-a41b1f584fb0" containerName="istio-proxy" Oct 30 18:34:26.976429 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:26.976430 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d98345b-38d7-49b3-8092-a41b1f584fb0" containerName="istio-proxy" Oct 30 18:34:26.976531 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:26.976463 4455 cpu_manager.go:414] "containerMap: removing container" podUID="7894da9d-a98c-45ac-9968-bfe4e9747c49" containerName="istio-proxy" Oct 30 18:34:26.976531 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:26.976469 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7894da9d-a98c-45ac-9968-bfe4e9747c49" containerName="istio-proxy" Oct 30 18:34:26.976531 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:26.976492 4455 cpu_manager.go:414] "containerMap: removing container" podUID="b0dbf354-3013-4694-b003-a55b1ddbaea0" containerName="maas-api" Oct 30 18:34:26.976531 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:26.976499 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0dbf354-3013-4694-b003-a55b1ddbaea0" containerName="maas-api" Oct 30 18:34:26.976661 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:26.976616 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0dbf354-3013-4694-b003-a55b1ddbaea0" containerName="maas-api" Oct 30 18:34:26.976661 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:26.976630 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d98345b-38d7-49b3-8092-a41b1f584fb0" containerName="istio-proxy" Oct 30 18:34:26.976661 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:26.976641 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="7894da9d-a98c-45ac-9968-bfe4e9747c49" containerName="istio-proxy" Oct 30 18:34:26.980707 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:26.980690 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" Oct 30 18:34:27.090371 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:27.090237 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8a6e230-4b8f-4d8d-af42-d6660522152e-istio-token\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" Oct 30 18:34:27.090371 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:27.090352 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8a6e230-4b8f-4d8d-af42-d6660522152e-istio-podinfo\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" Oct 30 18:34:27.090665 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:27.090393 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8a6e230-4b8f-4d8d-af42-d6660522152e-istio-envoy\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" Oct 30 18:34:27.090665 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:27.090442 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a6e230-4b8f-4d8d-af42-d6660522152e-credential-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" Oct 30 18:34:27.090665 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:27.090475 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8a6e230-4b8f-4d8d-af42-d6660522152e-istiod-ca-cert\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" Oct 30 18:34:27.090665 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:27.090542 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a6e230-4b8f-4d8d-af42-d6660522152e-workload-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" Oct 30 18:34:27.090884 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:27.090687 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8a6e230-4b8f-4d8d-af42-d6660522152e-istio-data\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" Oct 30 18:34:27.090884 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:27.090749 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htc5m\" (UniqueName: \"kubernetes.io/projected/a8a6e230-4b8f-4d8d-af42-d6660522152e-kube-api-access-htc5m\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" Oct 30 18:34:27.090884 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:27.090827 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8a6e230-4b8f-4d8d-af42-d6660522152e-workload-certs\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" Oct 30 18:34:27.101931 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:27.099828 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6"] Oct 30 18:34:27.192466 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:27.192426 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8a6e230-4b8f-4d8d-af42-d6660522152e-workload-certs\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" Oct 30 18:34:27.192662 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:27.192612 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8a6e230-4b8f-4d8d-af42-d6660522152e-istio-token\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" Oct 30 18:34:27.192740 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:27.192691 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8a6e230-4b8f-4d8d-af42-d6660522152e-istio-podinfo\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" Oct 30 18:34:27.192740 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:27.192727 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8a6e230-4b8f-4d8d-af42-d6660522152e-istio-envoy\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" Oct 30 18:34:27.192866 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:27.192780 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a6e230-4b8f-4d8d-af42-d6660522152e-credential-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" Oct 30 18:34:27.192866 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:27.192816 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8a6e230-4b8f-4d8d-af42-d6660522152e-istiod-ca-cert\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" Oct 30 18:34:27.192974 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:27.192884 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a6e230-4b8f-4d8d-af42-d6660522152e-workload-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" Oct 30 18:34:27.192974 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:27.192902 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8a6e230-4b8f-4d8d-af42-d6660522152e-workload-certs\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" Oct 30 18:34:27.193089 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:27.193017 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8a6e230-4b8f-4d8d-af42-d6660522152e-istio-data\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" Oct 30 18:34:27.193089 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:27.193075 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htc5m\" (UniqueName: \"kubernetes.io/projected/a8a6e230-4b8f-4d8d-af42-d6660522152e-kube-api-access-htc5m\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" Oct 30 18:34:27.193196 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:27.193155 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a6e230-4b8f-4d8d-af42-d6660522152e-credential-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" Oct 30 18:34:27.194452 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:27.193585 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8a6e230-4b8f-4d8d-af42-d6660522152e-istio-data\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" Oct 30 18:34:27.194452 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:27.193864 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a6e230-4b8f-4d8d-af42-d6660522152e-workload-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" Oct 30 18:34:27.194452 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:27.193886 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8a6e230-4b8f-4d8d-af42-d6660522152e-istiod-ca-cert\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" Oct 30 18:34:27.196536 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:27.196505 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8a6e230-4b8f-4d8d-af42-d6660522152e-istio-envoy\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" Oct 30 18:34:27.196698 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:27.196672 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8a6e230-4b8f-4d8d-af42-d6660522152e-istio-podinfo\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" Oct 30 18:34:27.221339 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:27.221290 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8a6e230-4b8f-4d8d-af42-d6660522152e-istio-token\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" Oct 30 18:34:27.237292 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:27.237233 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htc5m\" (UniqueName: \"kubernetes.io/projected/a8a6e230-4b8f-4d8d-af42-d6660522152e-kube-api-access-htc5m\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" Oct 30 18:34:27.298677 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:27.298640 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" Oct 30 18:34:27.637338 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:34:27.637291 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8a6e230_4b8f_4d8d_af42_d6660522152e.slice/crio-416079477a8f875ad962365536d51229ae3c4bc8e49fb669cfe1d1f432753cf4 WatchSource:0}: Error finding container 416079477a8f875ad962365536d51229ae3c4bc8e49fb669cfe1d1f432753cf4: Status 404 returned error can't find the container with id 416079477a8f875ad962365536d51229ae3c4bc8e49fb669cfe1d1f432753cf4 Oct 30 18:34:27.640462 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:27.640436 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6"] Oct 30 18:34:27.640827 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:27.640794 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:34:27.640901 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:27.640873 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:34:27.640966 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:27.640912 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:34:28.216744 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:28.216701 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" event={"ID":"a8a6e230-4b8f-4d8d-af42-d6660522152e","Type":"ContainerStarted","Data":"e2002cc6bfa68049e67590ef9ea1a59f8c518482ed44f52c2ff7d730d7e55671"} Oct 30 18:34:28.217278 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:28.216805 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" event={"ID":"a8a6e230-4b8f-4d8d-af42-d6660522152e","Type":"ContainerStarted","Data":"416079477a8f875ad962365536d51229ae3c4bc8e49fb669cfe1d1f432753cf4"} Oct 30 18:34:28.284761 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:28.268572 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" podStartSLOduration=2.268553309 podStartE2EDuration="2.268553309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:34:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:34:28.265653944 +0000 UTC m=+2520.753290975" watchObservedRunningTime="2026-10-30 18:34:28.268553309 +0000 UTC m=+2520.756190343" Oct 30 18:34:28.299227 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:28.299102 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" Oct 30 18:34:28.301790 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:28.301750 4455 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 30 18:34:28.301969 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:28.301834 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" podUID="a8a6e230-4b8f-4d8d-af42-d6660522152e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 30 18:34:29.299981 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:29.299944 4455 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 30 18:34:29.300450 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:29.300002 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" podUID="a8a6e230-4b8f-4d8d-af42-d6660522152e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 30 18:34:30.298995 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:30.298956 4455 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 30 18:34:30.299226 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:30.299011 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" podUID="a8a6e230-4b8f-4d8d-af42-d6660522152e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 30 18:34:30.414938 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:34:30.414888 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:34:31.299107 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:31.299060 4455 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 30 18:34:31.299288 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:31.299130 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" podUID="a8a6e230-4b8f-4d8d-af42-d6660522152e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 30 18:34:32.299757 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:32.299716 4455 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 30 18:34:32.300499 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:32.300444 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" podUID="a8a6e230-4b8f-4d8d-af42-d6660522152e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 30 18:34:33.300201 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:33.299689 4455 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 30 18:34:33.300201 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:33.299757 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" podUID="a8a6e230-4b8f-4d8d-af42-d6660522152e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 30 18:34:34.409378 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:34.407932 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" Oct 30 18:34:34.409378 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:34.408822 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" Oct 30 18:34:34.412771 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:34:34.412738 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" Oct 30 18:35:00.391702 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:35:00.391666 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:35:01.017950 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:01.017915 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4"] Oct 30 18:35:01.023971 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:01.023944 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4" Oct 30 18:35:01.045695 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:01.045666 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-ac422f-serving-cert\"" Oct 30 18:35:01.050358 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:01.050310 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4"] Oct 30 18:35:01.099409 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:01.099373 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d0733d2-85ad-4966-9740-97fffe7f54f6-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4\" (UID: \"9d0733d2-85ad-4966-9740-97fffe7f54f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4" Oct 30 18:35:01.099409 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:01.099422 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d0733d2-85ad-4966-9740-97fffe7f54f6-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4\" (UID: \"9d0733d2-85ad-4966-9740-97fffe7f54f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4" Oct 30 18:35:01.099706 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:01.099453 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qq2q\" (UniqueName: \"kubernetes.io/projected/9d0733d2-85ad-4966-9740-97fffe7f54f6-kube-api-access-5qq2q\") pod \"maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4\" (UID: \"9d0733d2-85ad-4966-9740-97fffe7f54f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4" Oct 30 18:35:01.099706 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:01.099621 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d0733d2-85ad-4966-9740-97fffe7f54f6-metrics-certs\") pod \"maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4\" (UID: \"9d0733d2-85ad-4966-9740-97fffe7f54f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4" Oct 30 18:35:01.200963 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:01.200928 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d0733d2-85ad-4966-9740-97fffe7f54f6-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4\" (UID: \"9d0733d2-85ad-4966-9740-97fffe7f54f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4" Oct 30 18:35:01.200963 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:01.200973 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d0733d2-85ad-4966-9740-97fffe7f54f6-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4\" (UID: \"9d0733d2-85ad-4966-9740-97fffe7f54f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4" Oct 30 18:35:01.201250 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:01.200992 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qq2q\" (UniqueName: \"kubernetes.io/projected/9d0733d2-85ad-4966-9740-97fffe7f54f6-kube-api-access-5qq2q\") pod \"maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4\" (UID: \"9d0733d2-85ad-4966-9740-97fffe7f54f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4" Oct 30 18:35:01.201250 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:01.201091 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d0733d2-85ad-4966-9740-97fffe7f54f6-metrics-certs\") pod \"maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4\" (UID: \"9d0733d2-85ad-4966-9740-97fffe7f54f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4" Oct 30 18:35:01.201822 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:01.201791 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d0733d2-85ad-4966-9740-97fffe7f54f6-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4\" (UID: \"9d0733d2-85ad-4966-9740-97fffe7f54f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4" Oct 30 18:35:01.203839 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:01.203812 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d0733d2-85ad-4966-9740-97fffe7f54f6-metrics-certs\") pod \"maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4\" (UID: \"9d0733d2-85ad-4966-9740-97fffe7f54f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4" Oct 30 18:35:01.203945 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:01.203876 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d0733d2-85ad-4966-9740-97fffe7f54f6-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4\" (UID: \"9d0733d2-85ad-4966-9740-97fffe7f54f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4" Oct 30 18:35:01.235627 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:01.235588 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qq2q\" (UniqueName: \"kubernetes.io/projected/9d0733d2-85ad-4966-9740-97fffe7f54f6-kube-api-access-5qq2q\") pod \"maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4\" (UID: \"9d0733d2-85ad-4966-9740-97fffe7f54f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4" Oct 30 18:35:01.353756 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:01.353632 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4" Oct 30 18:35:01.718418 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:35:01.718386 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d0733d2_85ad_4966_9740_97fffe7f54f6.slice/crio-3cf887f97e49996f6122d9a7eff5a67c59dccb33f3da95b2935d1566647704cd WatchSource:0}: Error finding container 3cf887f97e49996f6122d9a7eff5a67c59dccb33f3da95b2935d1566647704cd: Status 404 returned error can't find the container with id 3cf887f97e49996f6122d9a7eff5a67c59dccb33f3da95b2935d1566647704cd Oct 30 18:35:01.718778 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:01.718609 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4"] Oct 30 18:35:01.719777 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:01.719758 4455 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:35:02.664941 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:02.664901 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4" event={"ID":"9d0733d2-85ad-4966-9740-97fffe7f54f6","Type":"ContainerStarted","Data":"8ed2643be90cff51f72e2ea02dfba6ff86cad80c591e675d79875db9b5f42f1e"} Oct 30 18:35:02.665124 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:02.664970 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4" Oct 30 18:35:02.665124 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:02.664992 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4" event={"ID":"9d0733d2-85ad-4966-9740-97fffe7f54f6","Type":"ContainerStarted","Data":"3cf887f97e49996f6122d9a7eff5a67c59dccb33f3da95b2935d1566647704cd"} Oct 30 18:35:02.712723 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:02.712608 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4" podStartSLOduration=2.273048057 podStartE2EDuration="2.712587731s" totalImagesPullingTime="439.539674ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:35:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:35:02.712520861 +0000 UTC m=+2555.200157900" watchObservedRunningTime="2026-10-30 18:35:02.712587731 +0000 UTC m=+2555.200224765" Oct 30 18:35:08.681048 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:08.680961 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4" Oct 30 18:35:30.394621 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:35:30.394579 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:35:36.687351 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:36.664194 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" podUID="38cd4356-6595-419d-a6f2-251106d67bc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.204:8000/ready\": context deadline exceeded" Oct 30 18:35:37.359744 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:37.359697 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:35:37.716004 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:37.715964 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" podUID="38cd4356-6595-419d-a6f2-251106d67bc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.204:8000/ready\": context deadline exceeded" Oct 30 18:35:37.993041 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:37.992600 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" Oct 30 18:35:38.372432 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:38.372007 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:38.372889 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:38.372866 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:35:38.837364 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:38.741220 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" podUID="38cd4356-6595-419d-a6f2-251106d67bc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.204:8000/ready\": context deadline exceeded" Oct 30 18:35:39.375524 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:39.375478 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:35:39.939801 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:39.908757 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" podUID="38cd4356-6595-419d-a6f2-251106d67bc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.204:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:39.966544 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:39.966499 4455 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:35:39.966817 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:39.966789 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" podUID="d2c2d808-5274-484e-8520-1f88fa79fc47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:40.403350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:40.402994 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:35:40.941354 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:40.940467 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" podUID="38cd4356-6595-419d-a6f2-251106d67bc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.204:8000/ready\": context deadline exceeded" Oct 30 18:35:41.210404 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:41.205519 4455 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:35:41.210404 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:41.205602 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" podUID="b933cb99-2e87-4707-a33b-de09b5c03536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:41.234439 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:41.234265 4455 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" podUID="38cd4356-6595-419d-a6f2-251106d67bc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.204:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:41.235891 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:41.235845 4455 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:35:41.236174 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:41.236147 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" podUID="1ee94ddb-0834-4ee6-835c-4363b672e1c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:41.431347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:41.423421 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:35:41.942468 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:41.942418 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" podUID="38cd4356-6595-419d-a6f2-251106d67bc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.204:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:42.535367 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:42.521621 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:35:42.943881 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:42.943768 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" podUID="38cd4356-6595-419d-a6f2-251106d67bc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.204:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:43.562778 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:43.562735 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:43.734357 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:43.733403 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:35:43.734357 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:43.733464 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" podUID="c7c21be6-b9eb-401c-aa3c-af5b62960cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:43.745554 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:43.745522 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:35:44.023783 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:44.023176 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" podUID="38cd4356-6595-419d-a6f2-251106d67bc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.204:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:44.097162 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:44.097128 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:35:44.097647 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:44.097527 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" podUID="4e7b0c73-0041-4549-b47e-76a7aaed2b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:44.868614 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:44.867235 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" Oct 30 18:35:59.488227 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:59.488189 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ac632e-75c75cf4cb-nvwzm"] Oct 30 18:35:59.524686 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:59.524653 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ac632e-7d5967f8dc-blb7b"] Oct 30 18:35:59.843868 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:59.842549 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg"] Oct 30 18:35:59.843868 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:35:59.842874 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg" podUID="37f249f2-67f0-4ba7-bb80-05d7c6e698e9" containerName="maas-api" containerID="cri-o://18aec4832db0ae88a6570d183a511bfebb79d187f793a6c8f78d81c09b999e5d" gracePeriod=30 Oct 30 18:36:00.164522 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:00.164480 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg" Oct 30 18:36:00.183196 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:00.183159 4455 generic.go:356] "Generic (PLEG): container finished" podID="37f249f2-67f0-4ba7-bb80-05d7c6e698e9" containerID="18aec4832db0ae88a6570d183a511bfebb79d187f793a6c8f78d81c09b999e5d" exitCode=0 Oct 30 18:36:00.183445 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:00.183221 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg" Oct 30 18:36:00.183445 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:00.183246 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg" event={"ID":"37f249f2-67f0-4ba7-bb80-05d7c6e698e9","Type":"ContainerDied","Data":"18aec4832db0ae88a6570d183a511bfebb79d187f793a6c8f78d81c09b999e5d"} Oct 30 18:36:00.183445 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:00.183314 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg" event={"ID":"37f249f2-67f0-4ba7-bb80-05d7c6e698e9","Type":"ContainerDied","Data":"882fd8299500d3835b4c3b376723bbf3e02642fe2099089f835cfbd21564de6a"} Oct 30 18:36:00.183445 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:00.183353 4455 scope.go:122] "RemoveContainer" containerID="18aec4832db0ae88a6570d183a511bfebb79d187f793a6c8f78d81c09b999e5d" Oct 30 18:36:00.197636 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:00.197608 4455 scope.go:122] "RemoveContainer" containerID="18aec4832db0ae88a6570d183a511bfebb79d187f793a6c8f78d81c09b999e5d" Oct 30 18:36:00.198003 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:36:00.197976 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18aec4832db0ae88a6570d183a511bfebb79d187f793a6c8f78d81c09b999e5d\": container with ID starting with 18aec4832db0ae88a6570d183a511bfebb79d187f793a6c8f78d81c09b999e5d not found: ID does not exist" containerID="18aec4832db0ae88a6570d183a511bfebb79d187f793a6c8f78d81c09b999e5d" Oct 30 18:36:00.198104 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:00.198011 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18aec4832db0ae88a6570d183a511bfebb79d187f793a6c8f78d81c09b999e5d"} err="failed to get container status \"18aec4832db0ae88a6570d183a511bfebb79d187f793a6c8f78d81c09b999e5d\": rpc error: code = NotFound desc = could not find container \"18aec4832db0ae88a6570d183a511bfebb79d187f793a6c8f78d81c09b999e5d\": container with ID starting with 18aec4832db0ae88a6570d183a511bfebb79d187f793a6c8f78d81c09b999e5d not found: ID does not exist" Oct 30 18:36:00.227758 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:00.227720 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37f249f2-67f0-4ba7-bb80-05d7c6e698e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/37f249f2-67f0-4ba7-bb80-05d7c6e698e9-metrics-certs\") pod \"37f249f2-67f0-4ba7-bb80-05d7c6e698e9\" (UID: \"37f249f2-67f0-4ba7-bb80-05d7c6e698e9\") " Oct 30 18:36:00.227951 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:00.227776 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37f249f2-67f0-4ba7-bb80-05d7c6e698e9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37f249f2-67f0-4ba7-bb80-05d7c6e698e9-maas-api-tls\") pod \"37f249f2-67f0-4ba7-bb80-05d7c6e698e9\" (UID: \"37f249f2-67f0-4ba7-bb80-05d7c6e698e9\") " Oct 30 18:36:00.227951 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:00.227892 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37f249f2-67f0-4ba7-bb80-05d7c6e698e9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37f249f2-67f0-4ba7-bb80-05d7c6e698e9-openshift-service-ca\") pod \"37f249f2-67f0-4ba7-bb80-05d7c6e698e9\" (UID: \"37f249f2-67f0-4ba7-bb80-05d7c6e698e9\") " Oct 30 18:36:00.228149 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:00.228132 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37f249f2-67f0-4ba7-bb80-05d7c6e698e9-kube-api-access-j47wv\" (UniqueName: \"kubernetes.io/projected/37f249f2-67f0-4ba7-bb80-05d7c6e698e9-kube-api-access-j47wv\") pod \"37f249f2-67f0-4ba7-bb80-05d7c6e698e9\" (UID: \"37f249f2-67f0-4ba7-bb80-05d7c6e698e9\") " Oct 30 18:36:00.231009 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:00.229665 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37f249f2-67f0-4ba7-bb80-05d7c6e698e9-openshift-service-ca" pod "37f249f2-67f0-4ba7-bb80-05d7c6e698e9" (UID: "37f249f2-67f0-4ba7-bb80-05d7c6e698e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:00.232534 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:00.232499 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f249f2-67f0-4ba7-bb80-05d7c6e698e9-kube-api-access-j47wv" pod "37f249f2-67f0-4ba7-bb80-05d7c6e698e9" (UID: "37f249f2-67f0-4ba7-bb80-05d7c6e698e9"). InnerVolumeSpecName "kube-api-access-j47wv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:00.239883 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:00.239787 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37f249f2-67f0-4ba7-bb80-05d7c6e698e9-metrics-certs" pod "37f249f2-67f0-4ba7-bb80-05d7c6e698e9" (UID: "37f249f2-67f0-4ba7-bb80-05d7c6e698e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:36:00.241480 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:00.241451 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37f249f2-67f0-4ba7-bb80-05d7c6e698e9-maas-api-tls" pod "37f249f2-67f0-4ba7-bb80-05d7c6e698e9" (UID: "37f249f2-67f0-4ba7-bb80-05d7c6e698e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:36:00.330473 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:00.330420 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37f249f2-67f0-4ba7-bb80-05d7c6e698e9-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:36:00.330473 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:00.330460 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37f249f2-67f0-4ba7-bb80-05d7c6e698e9-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:36:00.330473 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:00.330474 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37f249f2-67f0-4ba7-bb80-05d7c6e698e9-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:36:00.330473 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:00.330484 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j47wv\" (UniqueName: \"kubernetes.io/projected/37f249f2-67f0-4ba7-bb80-05d7c6e698e9-kube-api-access-j47wv\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:36:00.742343 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:36:00.738863 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:36:00.793283 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:00.793204 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg"] Oct 30 18:36:00.800571 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:00.800533 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ac632e-84d797f8f-cxbbg"] Oct 30 18:36:01.637059 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:01.637025 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37f249f2-67f0-4ba7-bb80-05d7c6e698e9" path="/var/lib/kubelet/pods/37f249f2-67f0-4ba7-bb80-05d7c6e698e9/volumes" Oct 30 18:36:04.489155 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.489113 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ac632e-75c75cf4cb-nvwzm" podUID="7be0ce35-cc15-480d-aaed-51c7a709b124" containerName="payload-processing" containerID="cri-o://b2468bdaba22afca15fa75643bdda8aa54241d4e4d0c1fc0d588229723462d57" gracePeriod=25 Oct 30 18:36:04.525276 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.525235 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ac632e-7d5967f8dc-blb7b" podUID="670e8cf0-8eb6-4480-821c-3ead1b573195" containerName="payload-pre-processing" containerID="cri-o://df350166a98e75c20eb18ef0b7c73d36e48b21c1b80a2b19da57751f94374fcf" gracePeriod=25 Oct 30 18:36:04.580411 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.580385 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7"] Oct 30 18:36:04.581566 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.581542 4455 cpu_manager.go:414] "containerMap: removing container" podUID="37f249f2-67f0-4ba7-bb80-05d7c6e698e9" containerName="maas-api" Oct 30 18:36:04.581566 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.581565 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37f249f2-67f0-4ba7-bb80-05d7c6e698e9" containerName="maas-api" Oct 30 18:36:04.581833 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.581807 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="37f249f2-67f0-4ba7-bb80-05d7c6e698e9" containerName="maas-api" Oct 30 18:36:04.586391 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.586366 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" Oct 30 18:36:04.626814 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.626779 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-e77b1d-openshift-default-dockercfg-k7wxj\"" Oct 30 18:36:04.644703 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.644681 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7"] Oct 30 18:36:04.652162 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.652016 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099a77ed-9f6a-4937-85c1-b7053bc602fa-istio-data\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" Oct 30 18:36:04.652162 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.652099 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099a77ed-9f6a-4937-85c1-b7053bc602fa-istio-podinfo\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" Oct 30 18:36:04.652425 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.652186 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099a77ed-9f6a-4937-85c1-b7053bc602fa-istio-envoy\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" Oct 30 18:36:04.652425 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.652215 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq25n\" (UniqueName: \"kubernetes.io/projected/099a77ed-9f6a-4937-85c1-b7053bc602fa-kube-api-access-fq25n\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" Oct 30 18:36:04.652425 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.652301 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099a77ed-9f6a-4937-85c1-b7053bc602fa-credential-socket\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" Oct 30 18:36:04.652581 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.652456 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099a77ed-9f6a-4937-85c1-b7053bc602fa-workload-socket\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" Oct 30 18:36:04.652700 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.652681 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099a77ed-9f6a-4937-85c1-b7053bc602fa-workload-certs\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" Oct 30 18:36:04.652826 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.652804 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099a77ed-9f6a-4937-85c1-b7053bc602fa-istiod-ca-cert\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" Oct 30 18:36:04.652890 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.652865 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099a77ed-9f6a-4937-85c1-b7053bc602fa-istio-token\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" Oct 30 18:36:04.756060 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.755978 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099a77ed-9f6a-4937-85c1-b7053bc602fa-istiod-ca-cert\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" Oct 30 18:36:04.756224 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.756073 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099a77ed-9f6a-4937-85c1-b7053bc602fa-istio-token\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" Oct 30 18:36:04.756286 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.756271 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099a77ed-9f6a-4937-85c1-b7053bc602fa-istio-data\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" Oct 30 18:36:04.756385 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.756314 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099a77ed-9f6a-4937-85c1-b7053bc602fa-istio-podinfo\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" Oct 30 18:36:04.756459 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.756407 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099a77ed-9f6a-4937-85c1-b7053bc602fa-istio-envoy\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" Oct 30 18:36:04.756459 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.756437 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq25n\" (UniqueName: \"kubernetes.io/projected/099a77ed-9f6a-4937-85c1-b7053bc602fa-kube-api-access-fq25n\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" Oct 30 18:36:04.756576 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.756476 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099a77ed-9f6a-4937-85c1-b7053bc602fa-credential-socket\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" Oct 30 18:36:04.756576 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.756547 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099a77ed-9f6a-4937-85c1-b7053bc602fa-workload-socket\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" Oct 30 18:36:04.756825 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.756807 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099a77ed-9f6a-4937-85c1-b7053bc602fa-workload-certs\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" Oct 30 18:36:04.757522 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.757341 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099a77ed-9f6a-4937-85c1-b7053bc602fa-workload-certs\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" Oct 30 18:36:04.758064 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.758040 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099a77ed-9f6a-4937-85c1-b7053bc602fa-istiod-ca-cert\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" Oct 30 18:36:04.759068 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.758556 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099a77ed-9f6a-4937-85c1-b7053bc602fa-credential-socket\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" Oct 30 18:36:04.759068 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.758891 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099a77ed-9f6a-4937-85c1-b7053bc602fa-workload-socket\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" Oct 30 18:36:04.762424 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.762397 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099a77ed-9f6a-4937-85c1-b7053bc602fa-istio-data\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" Oct 30 18:36:04.764812 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.764783 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099a77ed-9f6a-4937-85c1-b7053bc602fa-istio-podinfo\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" Oct 30 18:36:04.772029 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.772005 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099a77ed-9f6a-4937-85c1-b7053bc602fa-istio-envoy\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" Oct 30 18:36:04.799145 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.799103 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq25n\" (UniqueName: \"kubernetes.io/projected/099a77ed-9f6a-4937-85c1-b7053bc602fa-kube-api-access-fq25n\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" Oct 30 18:36:04.805009 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.804973 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099a77ed-9f6a-4937-85c1-b7053bc602fa-istio-token\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" Oct 30 18:36:04.834225 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.834182 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-293234-b68cdd699-dcj4f"] Oct 30 18:36:04.842042 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.842009 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-293234-b68cdd699-dcj4f" Oct 30 18:36:04.856784 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.856752 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-293234-dockercfg-4hvcw\"" Oct 30 18:36:04.862618 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.858595 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-293234\"" Oct 30 18:36:05.017401 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.975699 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-293234-b68cdd699-dcj4f"] Oct 30 18:36:05.017401 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.979835 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/877c2eae-69e9-42fe-bbb9-b7a8ced3fa06-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-293234-b68cdd699-dcj4f\" (UID: \"877c2eae-69e9-42fe-bbb9-b7a8ced3fa06\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-293234-b68cdd699-dcj4f" Oct 30 18:36:05.017401 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.979920 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttzzh\" (UniqueName: \"kubernetes.io/projected/877c2eae-69e9-42fe-bbb9-b7a8ced3fa06-kube-api-access-ttzzh\") pod \"payload-pre-processing-e2e-praxis-b-293234-b68cdd699-dcj4f\" (UID: \"877c2eae-69e9-42fe-bbb9-b7a8ced3fa06\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-293234-b68cdd699-dcj4f" Oct 30 18:36:05.017401 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:04.980301 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" Oct 30 18:36:05.243460 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.223945 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/877c2eae-69e9-42fe-bbb9-b7a8ced3fa06-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-293234-b68cdd699-dcj4f\" (UID: \"877c2eae-69e9-42fe-bbb9-b7a8ced3fa06\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-293234-b68cdd699-dcj4f" Oct 30 18:36:05.243460 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.224063 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttzzh\" (UniqueName: \"kubernetes.io/projected/877c2eae-69e9-42fe-bbb9-b7a8ced3fa06-kube-api-access-ttzzh\") pod \"payload-pre-processing-e2e-praxis-b-293234-b68cdd699-dcj4f\" (UID: \"877c2eae-69e9-42fe-bbb9-b7a8ced3fa06\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-293234-b68cdd699-dcj4f" Oct 30 18:36:05.243460 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.225940 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/877c2eae-69e9-42fe-bbb9-b7a8ced3fa06-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-293234-b68cdd699-dcj4f\" (UID: \"877c2eae-69e9-42fe-bbb9-b7a8ced3fa06\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-293234-b68cdd699-dcj4f" Oct 30 18:36:05.258130 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.258092 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-293234-75cd6bf784-f78v8"] Oct 30 18:36:05.269479 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.267859 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttzzh\" (UniqueName: \"kubernetes.io/projected/877c2eae-69e9-42fe-bbb9-b7a8ced3fa06-kube-api-access-ttzzh\") pod \"payload-pre-processing-e2e-praxis-b-293234-b68cdd699-dcj4f\" (UID: \"877c2eae-69e9-42fe-bbb9-b7a8ced3fa06\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-293234-b68cdd699-dcj4f" Oct 30 18:36:05.269479 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.268714 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ac632e-75c75cf4cb-nvwzm" Oct 30 18:36:05.275633 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.275602 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-293234-75cd6bf784-f78v8"] Oct 30 18:36:05.275827 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.275810 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-293234-75cd6bf784-f78v8" Oct 30 18:36:05.306052 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.305911 4455 generic.go:356] "Generic (PLEG): container finished" podID="7be0ce35-cc15-480d-aaed-51c7a709b124" containerID="b2468bdaba22afca15fa75643bdda8aa54241d4e4d0c1fc0d588229723462d57" exitCode=0 Oct 30 18:36:05.306052 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.306043 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ac632e-75c75cf4cb-nvwzm" Oct 30 18:36:05.306592 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.306143 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ac632e-75c75cf4cb-nvwzm" event={"ID":"7be0ce35-cc15-480d-aaed-51c7a709b124","Type":"ContainerDied","Data":"b2468bdaba22afca15fa75643bdda8aa54241d4e4d0c1fc0d588229723462d57"} Oct 30 18:36:05.306592 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.306174 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ac632e-75c75cf4cb-nvwzm" event={"ID":"7be0ce35-cc15-480d-aaed-51c7a709b124","Type":"ContainerDied","Data":"ba8f66b28ac92b3cf35596c4c9704abec45b23ca87f1752ac5d821f89f3f42d6"} Oct 30 18:36:05.306592 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.306195 4455 scope.go:122] "RemoveContainer" containerID="b2468bdaba22afca15fa75643bdda8aa54241d4e4d0c1fc0d588229723462d57" Oct 30 18:36:05.316128 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.316079 4455 generic.go:356] "Generic (PLEG): container finished" podID="670e8cf0-8eb6-4480-821c-3ead1b573195" containerID="df350166a98e75c20eb18ef0b7c73d36e48b21c1b80a2b19da57751f94374fcf" exitCode=0 Oct 30 18:36:05.316429 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.316388 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ac632e-7d5967f8dc-blb7b" event={"ID":"670e8cf0-8eb6-4480-821c-3ead1b573195","Type":"ContainerDied","Data":"df350166a98e75c20eb18ef0b7c73d36e48b21c1b80a2b19da57751f94374fcf"} Oct 30 18:36:05.318974 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.318893 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ac632e-7d5967f8dc-blb7b" Oct 30 18:36:05.328060 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.328017 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7be0ce35-cc15-480d-aaed-51c7a709b124-plugins-config-volume" pod "7be0ce35-cc15-480d-aaed-51c7a709b124" (UID: "7be0ce35-cc15-480d-aaed-51c7a709b124"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:05.328350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.328294 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7be0ce35-cc15-480d-aaed-51c7a709b124-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7be0ce35-cc15-480d-aaed-51c7a709b124-plugins-config-volume\") pod \"7be0ce35-cc15-480d-aaed-51c7a709b124\" (UID: \"7be0ce35-cc15-480d-aaed-51c7a709b124\") " Oct 30 18:36:05.328599 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.328577 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/670e8cf0-8eb6-4480-821c-3ead1b573195-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/670e8cf0-8eb6-4480-821c-3ead1b573195-plugins-config-volume\") pod \"670e8cf0-8eb6-4480-821c-3ead1b573195\" (UID: \"670e8cf0-8eb6-4480-821c-3ead1b573195\") " Oct 30 18:36:05.331608 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.329350 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/670e8cf0-8eb6-4480-821c-3ead1b573195-plugins-config-volume" pod "670e8cf0-8eb6-4480-821c-3ead1b573195" (UID: "670e8cf0-8eb6-4480-821c-3ead1b573195"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:05.331862 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.331821 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/670e8cf0-8eb6-4480-821c-3ead1b573195-kube-api-access-9vzf7\" (UniqueName: \"kubernetes.io/projected/670e8cf0-8eb6-4480-821c-3ead1b573195-kube-api-access-9vzf7\") pod \"670e8cf0-8eb6-4480-821c-3ead1b573195\" (UID: \"670e8cf0-8eb6-4480-821c-3ead1b573195\") " Oct 30 18:36:05.332058 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.332039 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7be0ce35-cc15-480d-aaed-51c7a709b124-kube-api-access-qtsxr\" (UniqueName: \"kubernetes.io/projected/7be0ce35-cc15-480d-aaed-51c7a709b124-kube-api-access-qtsxr\") pod \"7be0ce35-cc15-480d-aaed-51c7a709b124\" (UID: \"7be0ce35-cc15-480d-aaed-51c7a709b124\") " Oct 30 18:36:05.332679 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.332655 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxxnn\" (UniqueName: \"kubernetes.io/projected/fc61777b-475d-4137-91eb-56e488ece861-kube-api-access-rxxnn\") pod \"payload-processing-e2e-praxis-b-293234-75cd6bf784-f78v8\" (UID: \"fc61777b-475d-4137-91eb-56e488ece861\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-293234-75cd6bf784-f78v8" Oct 30 18:36:05.332985 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.332944 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc61777b-475d-4137-91eb-56e488ece861-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-293234-75cd6bf784-f78v8\" (UID: \"fc61777b-475d-4137-91eb-56e488ece861\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-293234-75cd6bf784-f78v8" Oct 30 18:36:05.333557 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.333536 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7be0ce35-cc15-480d-aaed-51c7a709b124-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:36:05.333707 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.333691 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/670e8cf0-8eb6-4480-821c-3ead1b573195-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:36:05.336092 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.336069 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/670e8cf0-8eb6-4480-821c-3ead1b573195-kube-api-access-9vzf7" pod "670e8cf0-8eb6-4480-821c-3ead1b573195" (UID: "670e8cf0-8eb6-4480-821c-3ead1b573195"). InnerVolumeSpecName "kube-api-access-9vzf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:05.337106 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.337074 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7be0ce35-cc15-480d-aaed-51c7a709b124-kube-api-access-qtsxr" pod "7be0ce35-cc15-480d-aaed-51c7a709b124" (UID: "7be0ce35-cc15-480d-aaed-51c7a709b124"). InnerVolumeSpecName "kube-api-access-qtsxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:05.342510 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.341222 4455 scope.go:122] "RemoveContainer" containerID="b2468bdaba22afca15fa75643bdda8aa54241d4e4d0c1fc0d588229723462d57" Oct 30 18:36:05.342510 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:36:05.341622 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2468bdaba22afca15fa75643bdda8aa54241d4e4d0c1fc0d588229723462d57\": container with ID starting with b2468bdaba22afca15fa75643bdda8aa54241d4e4d0c1fc0d588229723462d57 not found: ID does not exist" containerID="b2468bdaba22afca15fa75643bdda8aa54241d4e4d0c1fc0d588229723462d57" Oct 30 18:36:05.342510 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.341657 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2468bdaba22afca15fa75643bdda8aa54241d4e4d0c1fc0d588229723462d57"} err="failed to get container status \"b2468bdaba22afca15fa75643bdda8aa54241d4e4d0c1fc0d588229723462d57\": rpc error: code = NotFound desc = could not find container \"b2468bdaba22afca15fa75643bdda8aa54241d4e4d0c1fc0d588229723462d57\": container with ID starting with b2468bdaba22afca15fa75643bdda8aa54241d4e4d0c1fc0d588229723462d57 not found: ID does not exist" Oct 30 18:36:05.356872 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.356821 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-293234-b68cdd699-dcj4f" Oct 30 18:36:05.437281 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.435500 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxxnn\" (UniqueName: \"kubernetes.io/projected/fc61777b-475d-4137-91eb-56e488ece861-kube-api-access-rxxnn\") pod \"payload-processing-e2e-praxis-b-293234-75cd6bf784-f78v8\" (UID: \"fc61777b-475d-4137-91eb-56e488ece861\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-293234-75cd6bf784-f78v8" Oct 30 18:36:05.437281 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.435604 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc61777b-475d-4137-91eb-56e488ece861-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-293234-75cd6bf784-f78v8\" (UID: \"fc61777b-475d-4137-91eb-56e488ece861\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-293234-75cd6bf784-f78v8" Oct 30 18:36:05.437281 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.435994 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vzf7\" (UniqueName: \"kubernetes.io/projected/670e8cf0-8eb6-4480-821c-3ead1b573195-kube-api-access-9vzf7\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:36:05.437281 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.436019 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtsxr\" (UniqueName: \"kubernetes.io/projected/7be0ce35-cc15-480d-aaed-51c7a709b124-kube-api-access-qtsxr\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:36:05.437281 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.436925 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc61777b-475d-4137-91eb-56e488ece861-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-293234-75cd6bf784-f78v8\" (UID: \"fc61777b-475d-4137-91eb-56e488ece861\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-293234-75cd6bf784-f78v8" Oct 30 18:36:05.460211 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.460189 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxxnn\" (UniqueName: \"kubernetes.io/projected/fc61777b-475d-4137-91eb-56e488ece861-kube-api-access-rxxnn\") pod \"payload-processing-e2e-praxis-b-293234-75cd6bf784-f78v8\" (UID: \"fc61777b-475d-4137-91eb-56e488ece861\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-293234-75cd6bf784-f78v8" Oct 30 18:36:05.616779 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.616749 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-293234-75cd6bf784-f78v8" Oct 30 18:36:05.623917 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.623893 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7"] Oct 30 18:36:05.626611 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:36:05.626580 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod099a77ed_9f6a_4937_85c1_b7053bc602fa.slice/crio-083b64e5ddb789b80dccd98f1a02f460e2e8dcb1e72852f3ade88727c7a92a67 WatchSource:0}: Error finding container 083b64e5ddb789b80dccd98f1a02f460e2e8dcb1e72852f3ade88727c7a92a67: Status 404 returned error can't find the container with id 083b64e5ddb789b80dccd98f1a02f460e2e8dcb1e72852f3ade88727c7a92a67 Oct 30 18:36:05.632114 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.632073 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:36:05.632289 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.632176 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:36:05.632289 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.632229 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:36:05.662398 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.662276 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ac632e-75c75cf4cb-nvwzm"] Oct 30 18:36:05.673241 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.673211 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ac632e-75c75cf4cb-nvwzm"] Oct 30 18:36:05.709409 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.709008 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-293234-b68cdd699-dcj4f"] Oct 30 18:36:05.710531 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:36:05.710491 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod877c2eae_69e9_42fe_bbb9_b7a8ced3fa06.slice/crio-37a6bf4f25da988b9722357105847a47bfd79d0b616bf105f4d31b0284888554 WatchSource:0}: Error finding container 37a6bf4f25da988b9722357105847a47bfd79d0b616bf105f4d31b0284888554: Status 404 returned error can't find the container with id 37a6bf4f25da988b9722357105847a47bfd79d0b616bf105f4d31b0284888554 Oct 30 18:36:05.984846 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:05.984814 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-293234-75cd6bf784-f78v8"] Oct 30 18:36:05.985365 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:36:05.985304 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc61777b_475d_4137_91eb_56e488ece861.slice/crio-f5c533cafef578ab4800395664aa31522b0942880fb95738c1cf47fe8e88773e WatchSource:0}: Error finding container f5c533cafef578ab4800395664aa31522b0942880fb95738c1cf47fe8e88773e: Status 404 returned error can't find the container with id f5c533cafef578ab4800395664aa31522b0942880fb95738c1cf47fe8e88773e Oct 30 18:36:06.326929 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:06.326847 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ac632e-7d5967f8dc-blb7b" Oct 30 18:36:06.326929 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:06.326871 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ac632e-7d5967f8dc-blb7b" event={"ID":"670e8cf0-8eb6-4480-821c-3ead1b573195","Type":"ContainerDied","Data":"48ad4645ac95036c6f3051018f9b3401628db6584b36cfdcc4d1ed4aed0be985"} Oct 30 18:36:06.326929 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:06.326930 4455 scope.go:122] "RemoveContainer" containerID="df350166a98e75c20eb18ef0b7c73d36e48b21c1b80a2b19da57751f94374fcf" Oct 30 18:36:06.329049 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:06.329008 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-293234-b68cdd699-dcj4f" event={"ID":"877c2eae-69e9-42fe-bbb9-b7a8ced3fa06","Type":"ContainerStarted","Data":"8e6da89cad7efa940e420d81b1d841d83909a28ada3a01e72ec9fc267a9f0937"} Oct 30 18:36:06.329171 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:06.329064 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-293234-b68cdd699-dcj4f" event={"ID":"877c2eae-69e9-42fe-bbb9-b7a8ced3fa06","Type":"ContainerStarted","Data":"37a6bf4f25da988b9722357105847a47bfd79d0b616bf105f4d31b0284888554"} Oct 30 18:36:06.329171 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:06.329126 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-293234-b68cdd699-dcj4f" Oct 30 18:36:06.331076 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:06.331049 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" event={"ID":"099a77ed-9f6a-4937-85c1-b7053bc602fa","Type":"ContainerStarted","Data":"7ff97b59aafc88f59908b1532470bfaf7408787498b324608e2a5fb88b8d7d43"} Oct 30 18:36:06.331202 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:06.331107 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" event={"ID":"099a77ed-9f6a-4937-85c1-b7053bc602fa","Type":"ContainerStarted","Data":"083b64e5ddb789b80dccd98f1a02f460e2e8dcb1e72852f3ade88727c7a92a67"} Oct 30 18:36:06.332586 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:06.332563 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-293234-75cd6bf784-f78v8" event={"ID":"fc61777b-475d-4137-91eb-56e488ece861","Type":"ContainerStarted","Data":"7a3b9548e4d3e650b112866b33f876c355038dc134470904e5084534b5ce00ca"} Oct 30 18:36:06.332690 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:06.332605 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-293234-75cd6bf784-f78v8" event={"ID":"fc61777b-475d-4137-91eb-56e488ece861","Type":"ContainerStarted","Data":"f5c533cafef578ab4800395664aa31522b0942880fb95738c1cf47fe8e88773e"} Oct 30 18:36:06.332690 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:06.332635 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-293234-75cd6bf784-f78v8" Oct 30 18:36:06.390575 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:06.390532 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ac632e-7d5967f8dc-blb7b"] Oct 30 18:36:06.395844 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:06.395815 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ac632e-7d5967f8dc-blb7b"] Oct 30 18:36:06.437586 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:06.437526 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-293234-b68cdd699-dcj4f" podStartSLOduration=2.437506653 podStartE2EDuration="2.437506653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:06.434870117 +0000 UTC m=+2618.922507153" watchObservedRunningTime="2026-10-30 18:36:06.437506653 +0000 UTC m=+2618.925143685" Oct 30 18:36:06.653678 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:06.653536 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-293234-75cd6bf784-f78v8" podStartSLOduration=2.653515007 podStartE2EDuration="2.653515007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:06.556202519 +0000 UTC m=+2619.043839566" watchObservedRunningTime="2026-10-30 18:36:06.653515007 +0000 UTC m=+2619.141152043" Oct 30 18:36:06.659618 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:06.659557 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" podStartSLOduration=2.65953502 podStartE2EDuration="2.65953502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:06.501744926 +0000 UTC m=+2618.989381960" watchObservedRunningTime="2026-10-30 18:36:06.65953502 +0000 UTC m=+2619.147172071" Oct 30 18:36:07.005636 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:07.005597 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" Oct 30 18:36:07.007364 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:07.007317 4455 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 30 18:36:07.007533 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:07.007397 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" podUID="099a77ed-9f6a-4937-85c1-b7053bc602fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 30 18:36:07.642483 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:07.642442 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="670e8cf0-8eb6-4480-821c-3ead1b573195" path="/var/lib/kubelet/pods/670e8cf0-8eb6-4480-821c-3ead1b573195/volumes" Oct 30 18:36:07.643280 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:07.643254 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7be0ce35-cc15-480d-aaed-51c7a709b124" path="/var/lib/kubelet/pods/7be0ce35-cc15-480d-aaed-51c7a709b124/volumes" Oct 30 18:36:08.005167 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:08.005129 4455 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 30 18:36:08.005611 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:08.005192 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" podUID="099a77ed-9f6a-4937-85c1-b7053bc602fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 30 18:36:09.005456 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:09.005422 4455 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 30 18:36:09.005939 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:09.005492 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" podUID="099a77ed-9f6a-4937-85c1-b7053bc602fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 30 18:36:10.027413 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:10.025395 4455 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 30 18:36:10.027413 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:10.025448 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" podUID="099a77ed-9f6a-4937-85c1-b7053bc602fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 30 18:36:11.005103 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:11.005065 4455 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 30 18:36:11.005298 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:11.005133 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" podUID="099a77ed-9f6a-4937-85c1-b7053bc602fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 30 18:36:12.005714 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:12.005673 4455 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 30 18:36:12.006213 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:12.005737 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" podUID="099a77ed-9f6a-4937-85c1-b7053bc602fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 30 18:36:13.011163 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:13.011131 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" Oct 30 18:36:13.011686 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:13.011668 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" Oct 30 18:36:13.012610 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:13.012585 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" Oct 30 18:36:17.347587 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:17.347553 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-293234-b68cdd699-dcj4f" Oct 30 18:36:17.349193 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:17.349167 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-293234-75cd6bf784-f78v8" Oct 30 18:36:27.157671 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:27.157638 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck"] Oct 30 18:36:27.158531 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:27.158514 4455 cpu_manager.go:414] "containerMap: removing container" podUID="7be0ce35-cc15-480d-aaed-51c7a709b124" containerName="payload-processing" Oct 30 18:36:27.158531 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:27.158531 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7be0ce35-cc15-480d-aaed-51c7a709b124" containerName="payload-processing" Oct 30 18:36:27.158648 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:27.158575 4455 cpu_manager.go:414] "containerMap: removing container" podUID="670e8cf0-8eb6-4480-821c-3ead1b573195" containerName="payload-pre-processing" Oct 30 18:36:27.158648 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:27.158580 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="670e8cf0-8eb6-4480-821c-3ead1b573195" containerName="payload-pre-processing" Oct 30 18:36:27.158722 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:27.158704 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="670e8cf0-8eb6-4480-821c-3ead1b573195" containerName="payload-pre-processing" Oct 30 18:36:27.158753 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:27.158724 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="7be0ce35-cc15-480d-aaed-51c7a709b124" containerName="payload-processing" Oct 30 18:36:27.162721 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:27.162703 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck" Oct 30 18:36:27.191078 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:27.191032 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-293234-serving-cert\"" Oct 30 18:36:27.199139 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:27.199107 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck"] Oct 30 18:36:27.207468 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:27.207434 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1524899f-95c7-4831-a90b-e108b715a3b3-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck\" (UID: \"1524899f-95c7-4831-a90b-e108b715a3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck" Oct 30 18:36:27.207716 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:27.207659 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1524899f-95c7-4831-a90b-e108b715a3b3-metrics-certs\") pod \"maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck\" (UID: \"1524899f-95c7-4831-a90b-e108b715a3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck" Oct 30 18:36:27.207847 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:27.207742 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1524899f-95c7-4831-a90b-e108b715a3b3-maas-api-tls\") pod \"maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck\" (UID: \"1524899f-95c7-4831-a90b-e108b715a3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck" Oct 30 18:36:27.208121 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:27.208094 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v86fs\" (UniqueName: \"kubernetes.io/projected/1524899f-95c7-4831-a90b-e108b715a3b3-kube-api-access-v86fs\") pod \"maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck\" (UID: \"1524899f-95c7-4831-a90b-e108b715a3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck" Oct 30 18:36:27.310071 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:27.310035 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1524899f-95c7-4831-a90b-e108b715a3b3-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck\" (UID: \"1524899f-95c7-4831-a90b-e108b715a3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck" Oct 30 18:36:27.310071 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:27.310074 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1524899f-95c7-4831-a90b-e108b715a3b3-metrics-certs\") pod \"maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck\" (UID: \"1524899f-95c7-4831-a90b-e108b715a3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck" Oct 30 18:36:27.310370 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:27.310106 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1524899f-95c7-4831-a90b-e108b715a3b3-maas-api-tls\") pod \"maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck\" (UID: \"1524899f-95c7-4831-a90b-e108b715a3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck" Oct 30 18:36:27.310370 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:27.310289 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v86fs\" (UniqueName: \"kubernetes.io/projected/1524899f-95c7-4831-a90b-e108b715a3b3-kube-api-access-v86fs\") pod \"maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck\" (UID: \"1524899f-95c7-4831-a90b-e108b715a3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck" Oct 30 18:36:27.310370 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:36:27.310297 4455 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-293234-serving-cert: secret "maas-api-e2e-praxis-b-293234-serving-cert" not found Oct 30 18:36:27.310542 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:36:27.310387 4455 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1524899f-95c7-4831-a90b-e108b715a3b3-maas-api-tls podName:1524899f-95c7-4831-a90b-e108b715a3b3 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:27.810364282 +0000 UTC m=+2640.298001294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1524899f-95c7-4831-a90b-e108b715a3b3-maas-api-tls") pod "maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck" (UID: "1524899f-95c7-4831-a90b-e108b715a3b3") : secret "maas-api-e2e-praxis-b-293234-serving-cert" not found Oct 30 18:36:27.310872 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:27.310844 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1524899f-95c7-4831-a90b-e108b715a3b3-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck\" (UID: \"1524899f-95c7-4831-a90b-e108b715a3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck" Oct 30 18:36:27.313032 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:27.313003 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1524899f-95c7-4831-a90b-e108b715a3b3-metrics-certs\") pod \"maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck\" (UID: \"1524899f-95c7-4831-a90b-e108b715a3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck" Oct 30 18:36:27.344344 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:27.335681 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v86fs\" (UniqueName: \"kubernetes.io/projected/1524899f-95c7-4831-a90b-e108b715a3b3-kube-api-access-v86fs\") pod \"maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck\" (UID: \"1524899f-95c7-4831-a90b-e108b715a3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck" Oct 30 18:36:27.822181 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:27.822148 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1524899f-95c7-4831-a90b-e108b715a3b3-maas-api-tls\") pod \"maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck\" (UID: \"1524899f-95c7-4831-a90b-e108b715a3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck" Oct 30 18:36:27.824811 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:27.824787 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1524899f-95c7-4831-a90b-e108b715a3b3-maas-api-tls\") pod \"maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck\" (UID: \"1524899f-95c7-4831-a90b-e108b715a3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck" Oct 30 18:36:28.080907 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:28.080822 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck" Oct 30 18:36:28.418403 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:28.417477 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a2a10a-c7c86844-xd2qw"] Oct 30 18:36:28.519740 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:28.495683 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a2a10a-5f7758695-k2fz7"] Oct 30 18:36:28.519740 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:28.516427 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck"] Oct 30 18:36:28.597870 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:28.597830 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck" event={"ID":"1524899f-95c7-4831-a90b-e108b715a3b3","Type":"ContainerStarted","Data":"2217506b916a99e19113b72e6187e0b28046991448592b663e394412235a352b"} Oct 30 18:36:29.609069 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:29.609030 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck" event={"ID":"1524899f-95c7-4831-a90b-e108b715a3b3","Type":"ContainerStarted","Data":"b55cab462d6626fd11dbe4e012dabcd8995eea57c529b3755714f0a8b86eddea"} Oct 30 18:36:29.609623 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:29.609093 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck" Oct 30 18:36:29.662278 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:29.662215 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck" podStartSLOduration=2.164499249 podStartE2EDuration="2.662197486s" totalImagesPullingTime="497.698237ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:29.656301309 +0000 UTC m=+2642.143938385" watchObservedRunningTime="2026-10-30 18:36:29.662197486 +0000 UTC m=+2642.149834522" Oct 30 18:36:30.394901 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:36:30.394860 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:36:33.418274 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:33.418215 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a2a10a-c7c86844-xd2qw" podUID="774b05b3-b9f0-412d-a530-d962c7344079" containerName="payload-processing" containerID="cri-o://a33608f42c6220ace09efcde794bbfa9c19408b0cb66ead69225b9de64563154" gracePeriod=25 Oct 30 18:36:33.496239 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:33.496168 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a2a10a-5f7758695-k2fz7" podUID="e30ec8f2-cffa-4713-823c-ce00bbcae4ad" containerName="payload-pre-processing" containerID="cri-o://b77b744dd9e53475d4bcb10cb9c5be2ada9ffef5018275d48492e82b3cd1f3e9" gracePeriod=25 Oct 30 18:36:33.557050 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:36:33.557023 4455 logging.go:55] [core] [Channel #8598 SubChannel #8600]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.189:50052", ServerName: "10.134.0.189:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.189:50052: connect: connection refused" Oct 30 18:36:33.557050 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:36:33.557023 4455 logging.go:55] [core] [Channel #8599 SubChannel #8601]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.188:50052", ServerName: "10.134.0.188:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.188:50052: connect: connection refused" Oct 30 18:36:33.647167 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:33.646961 4455 generic.go:356] "Generic (PLEG): container finished" podID="e30ec8f2-cffa-4713-823c-ce00bbcae4ad" containerID="b77b744dd9e53475d4bcb10cb9c5be2ada9ffef5018275d48492e82b3cd1f3e9" exitCode=0 Oct 30 18:36:33.647167 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:33.647037 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a2a10a-5f7758695-k2fz7" event={"ID":"e30ec8f2-cffa-4713-823c-ce00bbcae4ad","Type":"ContainerDied","Data":"b77b744dd9e53475d4bcb10cb9c5be2ada9ffef5018275d48492e82b3cd1f3e9"} Oct 30 18:36:33.649958 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:33.649854 4455 generic.go:356] "Generic (PLEG): container finished" podID="774b05b3-b9f0-412d-a530-d962c7344079" containerID="a33608f42c6220ace09efcde794bbfa9c19408b0cb66ead69225b9de64563154" exitCode=0 Oct 30 18:36:33.649958 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:33.649902 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a2a10a-c7c86844-xd2qw" event={"ID":"774b05b3-b9f0-412d-a530-d962c7344079","Type":"ContainerDied","Data":"a33608f42c6220ace09efcde794bbfa9c19408b0cb66ead69225b9de64563154"} Oct 30 18:36:33.816782 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:33.816750 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a2a10a-c7c86844-xd2qw" Oct 30 18:36:33.819466 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:33.819430 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/774b05b3-b9f0-412d-a530-d962c7344079-kube-api-access-cbx6g\" (UniqueName: \"kubernetes.io/projected/774b05b3-b9f0-412d-a530-d962c7344079-kube-api-access-cbx6g\") pod \"774b05b3-b9f0-412d-a530-d962c7344079\" (UID: \"774b05b3-b9f0-412d-a530-d962c7344079\") " Oct 30 18:36:33.819901 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:33.819877 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/774b05b3-b9f0-412d-a530-d962c7344079-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/774b05b3-b9f0-412d-a530-d962c7344079-plugins-config-volume\") pod \"774b05b3-b9f0-412d-a530-d962c7344079\" (UID: \"774b05b3-b9f0-412d-a530-d962c7344079\") " Oct 30 18:36:33.820302 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:33.820260 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/774b05b3-b9f0-412d-a530-d962c7344079-plugins-config-volume" pod "774b05b3-b9f0-412d-a530-d962c7344079" (UID: "774b05b3-b9f0-412d-a530-d962c7344079"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:33.820727 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:33.820689 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/774b05b3-b9f0-412d-a530-d962c7344079-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:36:33.822269 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:33.822245 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/774b05b3-b9f0-412d-a530-d962c7344079-kube-api-access-cbx6g" pod "774b05b3-b9f0-412d-a530-d962c7344079" (UID: "774b05b3-b9f0-412d-a530-d962c7344079"). InnerVolumeSpecName "kube-api-access-cbx6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:33.847491 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:33.847461 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a2a10a-5f7758695-k2fz7" Oct 30 18:36:33.921523 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:33.921492 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e30ec8f2-cffa-4713-823c-ce00bbcae4ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e30ec8f2-cffa-4713-823c-ce00bbcae4ad-plugins-config-volume\") pod \"e30ec8f2-cffa-4713-823c-ce00bbcae4ad\" (UID: \"e30ec8f2-cffa-4713-823c-ce00bbcae4ad\") " Oct 30 18:36:33.921731 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:33.921703 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e30ec8f2-cffa-4713-823c-ce00bbcae4ad-kube-api-access-cqmp4\" (UniqueName: \"kubernetes.io/projected/e30ec8f2-cffa-4713-823c-ce00bbcae4ad-kube-api-access-cqmp4\") pod \"e30ec8f2-cffa-4713-823c-ce00bbcae4ad\" (UID: \"e30ec8f2-cffa-4713-823c-ce00bbcae4ad\") " Oct 30 18:36:33.921968 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:33.921935 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e30ec8f2-cffa-4713-823c-ce00bbcae4ad-plugins-config-volume" pod "e30ec8f2-cffa-4713-823c-ce00bbcae4ad" (UID: "e30ec8f2-cffa-4713-823c-ce00bbcae4ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:33.922288 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:33.922270 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbx6g\" (UniqueName: \"kubernetes.io/projected/774b05b3-b9f0-412d-a530-d962c7344079-kube-api-access-cbx6g\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:36:33.922392 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:33.922296 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e30ec8f2-cffa-4713-823c-ce00bbcae4ad-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:36:33.924221 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:33.924176 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e30ec8f2-cffa-4713-823c-ce00bbcae4ad-kube-api-access-cqmp4" pod "e30ec8f2-cffa-4713-823c-ce00bbcae4ad" (UID: "e30ec8f2-cffa-4713-823c-ce00bbcae4ad"). InnerVolumeSpecName "kube-api-access-cqmp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:34.024005 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:34.023974 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqmp4\" (UniqueName: \"kubernetes.io/projected/e30ec8f2-cffa-4713-823c-ce00bbcae4ad-kube-api-access-cqmp4\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:36:34.556989 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:34.556949 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a2a10a-c7c86844-xd2qw" podUID="774b05b3-b9f0-412d-a530-d962c7344079" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.189:50052\" within 1s: context deadline exceeded" Oct 30 18:36:34.557421 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:34.556954 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a2a10a-5f7758695-k2fz7" podUID="e30ec8f2-cffa-4713-823c-ce00bbcae4ad" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.188:50052\" within 1s: context deadline exceeded" Oct 30 18:36:34.658683 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:34.658658 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a2a10a-c7c86844-xd2qw" Oct 30 18:36:34.658872 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:34.658655 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a2a10a-c7c86844-xd2qw" event={"ID":"774b05b3-b9f0-412d-a530-d962c7344079","Type":"ContainerDied","Data":"495d0c5a8b38c5c975e2cdbc290e9e740e9bb1885e4039278dbdff1928843e74"} Oct 30 18:36:34.658872 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:34.658803 4455 scope.go:122] "RemoveContainer" containerID="a33608f42c6220ace09efcde794bbfa9c19408b0cb66ead69225b9de64563154" Oct 30 18:36:34.660312 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:34.660292 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a2a10a-5f7758695-k2fz7" Oct 30 18:36:34.660602 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:34.660353 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a2a10a-5f7758695-k2fz7" event={"ID":"e30ec8f2-cffa-4713-823c-ce00bbcae4ad","Type":"ContainerDied","Data":"37aac5d231db4206f559ad3dd2bbb23d311a701237caee440be6b24b9134942e"} Oct 30 18:36:34.673217 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:34.673194 4455 scope.go:122] "RemoveContainer" containerID="b77b744dd9e53475d4bcb10cb9c5be2ada9ffef5018275d48492e82b3cd1f3e9" Oct 30 18:36:34.714412 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:34.714346 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a2a10a-5f7758695-k2fz7"] Oct 30 18:36:34.723876 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:34.723853 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a2a10a-5f7758695-k2fz7"] Oct 30 18:36:34.750014 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:34.749988 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a2a10a-c7c86844-xd2qw"] Oct 30 18:36:34.756158 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:34.756137 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a2a10a-c7c86844-xd2qw"] Oct 30 18:36:35.625484 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:35.625458 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck" Oct 30 18:36:35.636244 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:35.636217 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="774b05b3-b9f0-412d-a530-d962c7344079" path="/var/lib/kubelet/pods/774b05b3-b9f0-412d-a530-d962c7344079/volumes" Oct 30 18:36:35.636709 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:35.636694 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e30ec8f2-cffa-4713-823c-ce00bbcae4ad" path="/var/lib/kubelet/pods/e30ec8f2-cffa-4713-823c-ce00bbcae4ad/volumes" Oct 30 18:36:40.149152 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:40.149063 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-5cb3d5-c86c6bb98-lb6hc"] Oct 30 18:36:40.196086 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:40.196056 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5cb3d5-558fb66579-zq2b6"] Oct 30 18:36:45.149869 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:45.149807 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5cb3d5-c86c6bb98-lb6hc" podUID="abd1468f-513d-4fcb-8050-1b60650f28a6" containerName="payload-processing" containerID="cri-o://0f07d3b5fc074cfaef897696e6eba666a43918e2d7a60207657617304785d6c3" gracePeriod=25 Oct 30 18:36:45.197365 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:45.197306 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5cb3d5-558fb66579-zq2b6" podUID="2dd55385-0d97-47c9-9cc8-82659f695458" containerName="payload-pre-processing" containerID="cri-o://79bde611ee704de054b481446d7ca01f0c5bf796d1ec95659cca7ebbc4ec9136" gracePeriod=25 Oct 30 18:36:45.528083 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:45.528055 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5cb3d5-c86c6bb98-lb6hc" Oct 30 18:36:45.566162 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:45.566134 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5cb3d5-558fb66579-zq2b6" Oct 30 18:36:45.720714 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:45.720671 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dd55385-0d97-47c9-9cc8-82659f695458-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd55385-0d97-47c9-9cc8-82659f695458-plugins-config-volume\") pod \"2dd55385-0d97-47c9-9cc8-82659f695458\" (UID: \"2dd55385-0d97-47c9-9cc8-82659f695458\") " Oct 30 18:36:45.720917 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:45.720726 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abd1468f-513d-4fcb-8050-1b60650f28a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd1468f-513d-4fcb-8050-1b60650f28a6-plugins-config-volume\") pod \"abd1468f-513d-4fcb-8050-1b60650f28a6\" (UID: \"abd1468f-513d-4fcb-8050-1b60650f28a6\") " Oct 30 18:36:45.721032 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:45.721009 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd1468f-513d-4fcb-8050-1b60650f28a6-kube-api-access-j4hs2\" (UniqueName: \"kubernetes.io/projected/abd1468f-513d-4fcb-8050-1b60650f28a6-kube-api-access-j4hs2\") pod \"abd1468f-513d-4fcb-8050-1b60650f28a6\" (UID: \"abd1468f-513d-4fcb-8050-1b60650f28a6\") " Oct 30 18:36:45.721148 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:45.721122 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abd1468f-513d-4fcb-8050-1b60650f28a6-plugins-config-volume" pod "abd1468f-513d-4fcb-8050-1b60650f28a6" (UID: "abd1468f-513d-4fcb-8050-1b60650f28a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:45.721148 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:45.721130 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd55385-0d97-47c9-9cc8-82659f695458-plugins-config-volume" pod "2dd55385-0d97-47c9-9cc8-82659f695458" (UID: "2dd55385-0d97-47c9-9cc8-82659f695458"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:45.721305 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:45.721179 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dd55385-0d97-47c9-9cc8-82659f695458-kube-api-access-fdpft\" (UniqueName: \"kubernetes.io/projected/2dd55385-0d97-47c9-9cc8-82659f695458-kube-api-access-fdpft\") pod \"2dd55385-0d97-47c9-9cc8-82659f695458\" (UID: \"2dd55385-0d97-47c9-9cc8-82659f695458\") " Oct 30 18:36:45.722241 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:45.722212 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd55385-0d97-47c9-9cc8-82659f695458-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:36:45.722376 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:45.722246 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd1468f-513d-4fcb-8050-1b60650f28a6-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:36:45.724995 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:45.723874 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd55385-0d97-47c9-9cc8-82659f695458-kube-api-access-fdpft" pod "2dd55385-0d97-47c9-9cc8-82659f695458" (UID: "2dd55385-0d97-47c9-9cc8-82659f695458"). InnerVolumeSpecName "kube-api-access-fdpft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:45.725276 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:45.725254 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd1468f-513d-4fcb-8050-1b60650f28a6-kube-api-access-j4hs2" pod "abd1468f-513d-4fcb-8050-1b60650f28a6" (UID: "abd1468f-513d-4fcb-8050-1b60650f28a6"). InnerVolumeSpecName "kube-api-access-j4hs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:45.783477 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:45.783439 4455 generic.go:356] "Generic (PLEG): container finished" podID="2dd55385-0d97-47c9-9cc8-82659f695458" containerID="79bde611ee704de054b481446d7ca01f0c5bf796d1ec95659cca7ebbc4ec9136" exitCode=0 Oct 30 18:36:45.783676 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:45.783487 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5cb3d5-558fb66579-zq2b6" Oct 30 18:36:45.783676 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:45.783529 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5cb3d5-558fb66579-zq2b6" event={"ID":"2dd55385-0d97-47c9-9cc8-82659f695458","Type":"ContainerDied","Data":"79bde611ee704de054b481446d7ca01f0c5bf796d1ec95659cca7ebbc4ec9136"} Oct 30 18:36:45.783676 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:45.783588 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5cb3d5-558fb66579-zq2b6" event={"ID":"2dd55385-0d97-47c9-9cc8-82659f695458","Type":"ContainerDied","Data":"93574f55e506747c761d4d02b91016aa3dce9436e4a1148c84a24dba67b1fa13"} Oct 30 18:36:45.783676 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:45.783621 4455 scope.go:122] "RemoveContainer" containerID="79bde611ee704de054b481446d7ca01f0c5bf796d1ec95659cca7ebbc4ec9136" Oct 30 18:36:45.785412 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:45.785299 4455 generic.go:356] "Generic (PLEG): container finished" podID="abd1468f-513d-4fcb-8050-1b60650f28a6" containerID="0f07d3b5fc074cfaef897696e6eba666a43918e2d7a60207657617304785d6c3" exitCode=0 Oct 30 18:36:45.785412 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:45.785357 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5cb3d5-c86c6bb98-lb6hc" Oct 30 18:36:45.785412 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:45.785394 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5cb3d5-c86c6bb98-lb6hc" event={"ID":"abd1468f-513d-4fcb-8050-1b60650f28a6","Type":"ContainerDied","Data":"0f07d3b5fc074cfaef897696e6eba666a43918e2d7a60207657617304785d6c3"} Oct 30 18:36:45.785674 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:45.785437 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5cb3d5-c86c6bb98-lb6hc" event={"ID":"abd1468f-513d-4fcb-8050-1b60650f28a6","Type":"ContainerDied","Data":"af930a29b4b9ba99ae748aa9032d268f948b5cdd7c34b07f97ca39e3cd16a22e"} Oct 30 18:36:45.798553 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:45.798536 4455 scope.go:122] "RemoveContainer" containerID="79bde611ee704de054b481446d7ca01f0c5bf796d1ec95659cca7ebbc4ec9136" Oct 30 18:36:45.799044 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:36:45.798896 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79bde611ee704de054b481446d7ca01f0c5bf796d1ec95659cca7ebbc4ec9136\": container with ID starting with 79bde611ee704de054b481446d7ca01f0c5bf796d1ec95659cca7ebbc4ec9136 not found: ID does not exist" containerID="79bde611ee704de054b481446d7ca01f0c5bf796d1ec95659cca7ebbc4ec9136" Oct 30 18:36:45.799044 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:45.798936 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79bde611ee704de054b481446d7ca01f0c5bf796d1ec95659cca7ebbc4ec9136"} err="failed to get container status \"79bde611ee704de054b481446d7ca01f0c5bf796d1ec95659cca7ebbc4ec9136\": rpc error: code = NotFound desc = could not find container \"79bde611ee704de054b481446d7ca01f0c5bf796d1ec95659cca7ebbc4ec9136\": container with ID starting with 79bde611ee704de054b481446d7ca01f0c5bf796d1ec95659cca7ebbc4ec9136 not found: ID does not exist" Oct 30 18:36:45.799044 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:45.798958 4455 scope.go:122] "RemoveContainer" containerID="0f07d3b5fc074cfaef897696e6eba666a43918e2d7a60207657617304785d6c3" Oct 30 18:36:45.811884 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:45.811865 4455 scope.go:122] "RemoveContainer" containerID="0f07d3b5fc074cfaef897696e6eba666a43918e2d7a60207657617304785d6c3" Oct 30 18:36:45.812193 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:36:45.812171 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f07d3b5fc074cfaef897696e6eba666a43918e2d7a60207657617304785d6c3\": container with ID starting with 0f07d3b5fc074cfaef897696e6eba666a43918e2d7a60207657617304785d6c3 not found: ID does not exist" containerID="0f07d3b5fc074cfaef897696e6eba666a43918e2d7a60207657617304785d6c3" Oct 30 18:36:45.812271 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:45.812203 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f07d3b5fc074cfaef897696e6eba666a43918e2d7a60207657617304785d6c3"} err="failed to get container status \"0f07d3b5fc074cfaef897696e6eba666a43918e2d7a60207657617304785d6c3\": rpc error: code = NotFound desc = could not find container \"0f07d3b5fc074cfaef897696e6eba666a43918e2d7a60207657617304785d6c3\": container with ID starting with 0f07d3b5fc074cfaef897696e6eba666a43918e2d7a60207657617304785d6c3 not found: ID does not exist" Oct 30 18:36:45.823188 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:45.823165 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4hs2\" (UniqueName: \"kubernetes.io/projected/abd1468f-513d-4fcb-8050-1b60650f28a6-kube-api-access-j4hs2\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:36:45.823188 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:45.823189 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdpft\" (UniqueName: \"kubernetes.io/projected/2dd55385-0d97-47c9-9cc8-82659f695458-kube-api-access-fdpft\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:36:45.835892 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:45.835869 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-5cb3d5-c86c6bb98-lb6hc"] Oct 30 18:36:45.840590 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:45.840567 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-5cb3d5-c86c6bb98-lb6hc"] Oct 30 18:36:45.882393 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:45.882334 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5cb3d5-558fb66579-zq2b6"] Oct 30 18:36:45.889536 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:45.889507 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5cb3d5-558fb66579-zq2b6"] Oct 30 18:36:46.834120 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:46.834085 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq"] Oct 30 18:36:47.637543 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:47.637513 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dd55385-0d97-47c9-9cc8-82659f695458" path="/var/lib/kubelet/pods/2dd55385-0d97-47c9-9cc8-82659f695458/volumes" Oct 30 18:36:47.638027 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:47.638013 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abd1468f-513d-4fcb-8050-1b60650f28a6" path="/var/lib/kubelet/pods/abd1468f-513d-4fcb-8050-1b60650f28a6/volumes" Oct 30 18:36:51.435934 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:51.435897 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq"] Oct 30 18:36:51.436502 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:51.436173 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq" podUID="9fcbeab7-58a0-431a-87e7-e1a4fac339c3" containerName="maas-api" containerID="cri-o://181d91245914f016c05773b5b66c2423336f6d83f67d6336029aaecb69af4227" gracePeriod=30 Oct 30 18:36:51.897552 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:51.897257 4455 generic.go:356] "Generic (PLEG): container finished" podID="9fcbeab7-58a0-431a-87e7-e1a4fac339c3" containerID="181d91245914f016c05773b5b66c2423336f6d83f67d6336029aaecb69af4227" exitCode=0 Oct 30 18:36:51.897552 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:51.897483 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq" event={"ID":"9fcbeab7-58a0-431a-87e7-e1a4fac339c3","Type":"ContainerDied","Data":"181d91245914f016c05773b5b66c2423336f6d83f67d6336029aaecb69af4227"} Oct 30 18:36:52.070890 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:52.070843 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq" Oct 30 18:36:52.144010 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:52.143941 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fcbeab7-58a0-431a-87e7-e1a4fac339c3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fcbeab7-58a0-431a-87e7-e1a4fac339c3-openshift-service-ca\") pod \"9fcbeab7-58a0-431a-87e7-e1a4fac339c3\" (UID: \"9fcbeab7-58a0-431a-87e7-e1a4fac339c3\") " Oct 30 18:36:52.144363 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:52.144315 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fcbeab7-58a0-431a-87e7-e1a4fac339c3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fcbeab7-58a0-431a-87e7-e1a4fac339c3-maas-api-tls\") pod \"9fcbeab7-58a0-431a-87e7-e1a4fac339c3\" (UID: \"9fcbeab7-58a0-431a-87e7-e1a4fac339c3\") " Oct 30 18:36:52.145038 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:52.145007 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fcbeab7-58a0-431a-87e7-e1a4fac339c3-kube-api-access-6zzv5\" (UniqueName: \"kubernetes.io/projected/9fcbeab7-58a0-431a-87e7-e1a4fac339c3-kube-api-access-6zzv5\") pod \"9fcbeab7-58a0-431a-87e7-e1a4fac339c3\" (UID: \"9fcbeab7-58a0-431a-87e7-e1a4fac339c3\") " Oct 30 18:36:52.145271 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:52.145249 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fcbeab7-58a0-431a-87e7-e1a4fac339c3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fcbeab7-58a0-431a-87e7-e1a4fac339c3-metrics-certs\") pod \"9fcbeab7-58a0-431a-87e7-e1a4fac339c3\" (UID: \"9fcbeab7-58a0-431a-87e7-e1a4fac339c3\") " Oct 30 18:36:52.153658 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:52.150695 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fcbeab7-58a0-431a-87e7-e1a4fac339c3-openshift-service-ca" pod "9fcbeab7-58a0-431a-87e7-e1a4fac339c3" (UID: "9fcbeab7-58a0-431a-87e7-e1a4fac339c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:52.197265 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:52.197218 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fcbeab7-58a0-431a-87e7-e1a4fac339c3-metrics-certs" pod "9fcbeab7-58a0-431a-87e7-e1a4fac339c3" (UID: "9fcbeab7-58a0-431a-87e7-e1a4fac339c3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:36:52.197647 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:52.197626 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fcbeab7-58a0-431a-87e7-e1a4fac339c3-maas-api-tls" pod "9fcbeab7-58a0-431a-87e7-e1a4fac339c3" (UID: "9fcbeab7-58a0-431a-87e7-e1a4fac339c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:36:52.211595 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:52.211530 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fcbeab7-58a0-431a-87e7-e1a4fac339c3-kube-api-access-6zzv5" pod "9fcbeab7-58a0-431a-87e7-e1a4fac339c3" (UID: "9fcbeab7-58a0-431a-87e7-e1a4fac339c3"). InnerVolumeSpecName "kube-api-access-6zzv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:52.248342 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:52.248295 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fcbeab7-58a0-431a-87e7-e1a4fac339c3-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:36:52.248697 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:52.248671 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fcbeab7-58a0-431a-87e7-e1a4fac339c3-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:36:52.248838 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:52.248825 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fcbeab7-58a0-431a-87e7-e1a4fac339c3-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:36:52.248942 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:52.248931 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zzv5\" (UniqueName: \"kubernetes.io/projected/9fcbeab7-58a0-431a-87e7-e1a4fac339c3-kube-api-access-6zzv5\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:36:52.909036 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:52.909002 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq" Oct 30 18:36:52.909652 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:52.909025 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq" event={"ID":"9fcbeab7-58a0-431a-87e7-e1a4fac339c3","Type":"ContainerDied","Data":"6170b4a47b6c992c639e03049fc4f28e0238c065331ebdbd5f4d03ac9ed6d4ad"} Oct 30 18:36:52.909652 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:52.909105 4455 scope.go:122] "RemoveContainer" containerID="181d91245914f016c05773b5b66c2423336f6d83f67d6336029aaecb69af4227" Oct 30 18:36:52.963759 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:52.963723 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq"] Oct 30 18:36:52.975400 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:52.975367 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a2a10a-747955d96c-g67dq"] Oct 30 18:36:53.639816 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:53.639783 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fcbeab7-58a0-431a-87e7-e1a4fac339c3" path="/var/lib/kubelet/pods/9fcbeab7-58a0-431a-87e7-e1a4fac339c3/volumes" Oct 30 18:36:53.914198 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:53.914110 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-58d08b-866bd6bfff-8vjgc"] Oct 30 18:36:53.915494 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:53.915468 4455 cpu_manager.go:414] "containerMap: removing container" podUID="abd1468f-513d-4fcb-8050-1b60650f28a6" containerName="payload-processing" Oct 30 18:36:53.915494 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:53.915495 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abd1468f-513d-4fcb-8050-1b60650f28a6" containerName="payload-processing" Oct 30 18:36:53.915700 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:53.915512 4455 cpu_manager.go:414] "containerMap: removing container" podUID="9fcbeab7-58a0-431a-87e7-e1a4fac339c3" containerName="maas-api" Oct 30 18:36:53.915700 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:53.915521 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fcbeab7-58a0-431a-87e7-e1a4fac339c3" containerName="maas-api" Oct 30 18:36:53.915700 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:53.915555 4455 cpu_manager.go:414] "containerMap: removing container" podUID="774b05b3-b9f0-412d-a530-d962c7344079" containerName="payload-processing" Oct 30 18:36:53.915700 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:53.915563 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="774b05b3-b9f0-412d-a530-d962c7344079" containerName="payload-processing" Oct 30 18:36:53.915700 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:53.915609 4455 cpu_manager.go:414] "containerMap: removing container" podUID="e30ec8f2-cffa-4713-823c-ce00bbcae4ad" containerName="payload-pre-processing" Oct 30 18:36:53.915700 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:53.915617 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e30ec8f2-cffa-4713-823c-ce00bbcae4ad" containerName="payload-pre-processing" Oct 30 18:36:53.915700 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:53.915631 4455 cpu_manager.go:414] "containerMap: removing container" podUID="2dd55385-0d97-47c9-9cc8-82659f695458" containerName="payload-pre-processing" Oct 30 18:36:53.915700 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:53.915638 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dd55385-0d97-47c9-9cc8-82659f695458" containerName="payload-pre-processing" Oct 30 18:36:53.915947 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:53.915827 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fcbeab7-58a0-431a-87e7-e1a4fac339c3" containerName="maas-api" Oct 30 18:36:53.915947 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:53.915845 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="774b05b3-b9f0-412d-a530-d962c7344079" containerName="payload-processing" Oct 30 18:36:53.915947 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:53.915858 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="e30ec8f2-cffa-4713-823c-ce00bbcae4ad" containerName="payload-pre-processing" Oct 30 18:36:53.915947 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:53.915866 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dd55385-0d97-47c9-9cc8-82659f695458" containerName="payload-pre-processing" Oct 30 18:36:53.915947 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:53.915882 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="abd1468f-513d-4fcb-8050-1b60650f28a6" containerName="payload-processing" Oct 30 18:36:53.921794 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:53.921765 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-58d08b-866bd6bfff-8vjgc" Oct 30 18:36:53.994856 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:53.994821 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-58d08b\"" Oct 30 18:36:53.997020 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:53.995171 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-58d08b-dockercfg-l6xc8\"" Oct 30 18:36:54.044240 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:54.044201 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-58d08b-866bd6bfff-8vjgc"] Oct 30 18:36:54.097355 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:54.094696 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv5zg\" (UniqueName: \"kubernetes.io/projected/baac9ab5-2caa-4596-a802-46062f6fcb4c-kube-api-access-pv5zg\") pod \"payload-pre-processing-e2e-autores-58d08b-866bd6bfff-8vjgc\" (UID: \"baac9ab5-2caa-4596-a802-46062f6fcb4c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-58d08b-866bd6bfff-8vjgc" Oct 30 18:36:54.097355 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:54.094840 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baac9ab5-2caa-4596-a802-46062f6fcb4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-58d08b-866bd6bfff-8vjgc\" (UID: \"baac9ab5-2caa-4596-a802-46062f6fcb4c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-58d08b-866bd6bfff-8vjgc" Oct 30 18:36:54.160358 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:54.144399 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-58d08b-c58b968db-9f7zq"] Oct 30 18:36:54.160358 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:54.158046 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-58d08b-c58b968db-9f7zq" Oct 30 18:36:54.198270 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:54.196701 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv5zg\" (UniqueName: \"kubernetes.io/projected/baac9ab5-2caa-4596-a802-46062f6fcb4c-kube-api-access-pv5zg\") pod \"payload-pre-processing-e2e-autores-58d08b-866bd6bfff-8vjgc\" (UID: \"baac9ab5-2caa-4596-a802-46062f6fcb4c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-58d08b-866bd6bfff-8vjgc" Oct 30 18:36:54.198270 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:54.196848 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baac9ab5-2caa-4596-a802-46062f6fcb4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-58d08b-866bd6bfff-8vjgc\" (UID: \"baac9ab5-2caa-4596-a802-46062f6fcb4c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-58d08b-866bd6bfff-8vjgc" Oct 30 18:36:54.198270 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:54.197901 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baac9ab5-2caa-4596-a802-46062f6fcb4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-58d08b-866bd6bfff-8vjgc\" (UID: \"baac9ab5-2caa-4596-a802-46062f6fcb4c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-58d08b-866bd6bfff-8vjgc" Oct 30 18:36:54.275033 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:54.274977 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-58d08b-c58b968db-9f7zq"] Oct 30 18:36:54.282092 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:54.282056 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv5zg\" (UniqueName: \"kubernetes.io/projected/baac9ab5-2caa-4596-a802-46062f6fcb4c-kube-api-access-pv5zg\") pod \"payload-pre-processing-e2e-autores-58d08b-866bd6bfff-8vjgc\" (UID: \"baac9ab5-2caa-4596-a802-46062f6fcb4c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-58d08b-866bd6bfff-8vjgc" Oct 30 18:36:54.298336 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:54.298266 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03cfd14d-26f3-4294-a58d-9e47ad8b1b67-plugins-config-volume\") pod \"payload-processing-e2e-autores-58d08b-c58b968db-9f7zq\" (UID: \"03cfd14d-26f3-4294-a58d-9e47ad8b1b67\") " pod="openshift-ingress/payload-processing-e2e-autores-58d08b-c58b968db-9f7zq" Oct 30 18:36:54.298834 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:54.298807 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jllz4\" (UniqueName: \"kubernetes.io/projected/03cfd14d-26f3-4294-a58d-9e47ad8b1b67-kube-api-access-jllz4\") pod \"payload-processing-e2e-autores-58d08b-c58b968db-9f7zq\" (UID: \"03cfd14d-26f3-4294-a58d-9e47ad8b1b67\") " pod="openshift-ingress/payload-processing-e2e-autores-58d08b-c58b968db-9f7zq" Oct 30 18:36:54.301571 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:54.301552 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-58d08b-866bd6bfff-8vjgc" Oct 30 18:36:54.402028 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:54.401256 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jllz4\" (UniqueName: \"kubernetes.io/projected/03cfd14d-26f3-4294-a58d-9e47ad8b1b67-kube-api-access-jllz4\") pod \"payload-processing-e2e-autores-58d08b-c58b968db-9f7zq\" (UID: \"03cfd14d-26f3-4294-a58d-9e47ad8b1b67\") " pod="openshift-ingress/payload-processing-e2e-autores-58d08b-c58b968db-9f7zq" Oct 30 18:36:54.402028 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:54.401610 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03cfd14d-26f3-4294-a58d-9e47ad8b1b67-plugins-config-volume\") pod \"payload-processing-e2e-autores-58d08b-c58b968db-9f7zq\" (UID: \"03cfd14d-26f3-4294-a58d-9e47ad8b1b67\") " pod="openshift-ingress/payload-processing-e2e-autores-58d08b-c58b968db-9f7zq" Oct 30 18:36:54.405023 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:54.404237 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03cfd14d-26f3-4294-a58d-9e47ad8b1b67-plugins-config-volume\") pod \"payload-processing-e2e-autores-58d08b-c58b968db-9f7zq\" (UID: \"03cfd14d-26f3-4294-a58d-9e47ad8b1b67\") " pod="openshift-ingress/payload-processing-e2e-autores-58d08b-c58b968db-9f7zq" Oct 30 18:36:54.427063 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:54.426887 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jllz4\" (UniqueName: \"kubernetes.io/projected/03cfd14d-26f3-4294-a58d-9e47ad8b1b67-kube-api-access-jllz4\") pod \"payload-processing-e2e-autores-58d08b-c58b968db-9f7zq\" (UID: \"03cfd14d-26f3-4294-a58d-9e47ad8b1b67\") " pod="openshift-ingress/payload-processing-e2e-autores-58d08b-c58b968db-9f7zq" Oct 30 18:36:54.494159 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:54.494123 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-58d08b-c58b968db-9f7zq" Oct 30 18:36:54.764373 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:54.754667 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-58d08b-866bd6bfff-8vjgc"] Oct 30 18:36:55.206068 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:55.206027 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-58d08b-866bd6bfff-8vjgc" event={"ID":"baac9ab5-2caa-4596-a802-46062f6fcb4c","Type":"ContainerStarted","Data":"dc6587da2bd2eeecaa56d571929dacafed14ad5f11a9f5d8c2fcf18dca0fee1f"} Oct 30 18:36:55.238724 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:55.238695 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-58d08b-c58b968db-9f7zq"] Oct 30 18:36:56.273964 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:56.273927 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-58d08b-c58b968db-9f7zq" event={"ID":"03cfd14d-26f3-4294-a58d-9e47ad8b1b67","Type":"ContainerStarted","Data":"05433e05a88bd86a48bc23071a6045719e83df925ba2838a843b6225b977ee40"} Oct 30 18:36:56.274568 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:56.274541 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-58d08b-c58b968db-9f7zq" event={"ID":"03cfd14d-26f3-4294-a58d-9e47ad8b1b67","Type":"ContainerStarted","Data":"6c8865329862b160bc2d0f5ec37300ca06e5d31975a62bf741a12644737388a8"} Oct 30 18:36:56.275822 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:56.275801 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-58d08b-c58b968db-9f7zq" Oct 30 18:36:56.289362 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:56.289295 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-58d08b-866bd6bfff-8vjgc" event={"ID":"baac9ab5-2caa-4596-a802-46062f6fcb4c","Type":"ContainerStarted","Data":"712e1f01575964bb17183355ffc5ab13f402fa4d7a934c34ff2d92b9fc36e607"} Oct 30 18:36:56.290646 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:56.290621 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-58d08b-866bd6bfff-8vjgc" Oct 30 18:36:56.340813 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:56.340753 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-58d08b-c58b968db-9f7zq" podStartSLOduration=3.340733499 podStartE2EDuration="3.340733499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:56.319804916 +0000 UTC m=+2668.807441950" watchObservedRunningTime="2026-10-30 18:36:56.340733499 +0000 UTC m=+2668.828370534" Oct 30 18:36:56.364838 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:56.364701 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-58d08b-866bd6bfff-8vjgc" podStartSLOduration=3.364682305 podStartE2EDuration="3.364682305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:56.362103695 +0000 UTC m=+2668.849740752" watchObservedRunningTime="2026-10-30 18:36:56.364682305 +0000 UTC m=+2668.852319340" Oct 30 18:36:56.784668 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:56.784602 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m"] Oct 30 18:36:56.785117 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:56.785085 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m" podUID="b9a7536b-d288-4ea3-b1d1-f824ec859a47" containerName="maas-api" containerID="cri-o://c15d1d7108bd1881634d5fd0f048e1ff9131f51d65cdd24e710785d22c9813f3" gracePeriod=30 Oct 30 18:36:57.089576 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:57.089530 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m" Oct 30 18:36:57.196348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:57.193960 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9a7536b-d288-4ea3-b1d1-f824ec859a47-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9a7536b-d288-4ea3-b1d1-f824ec859a47-metrics-certs\") pod \"b9a7536b-d288-4ea3-b1d1-f824ec859a47\" (UID: \"b9a7536b-d288-4ea3-b1d1-f824ec859a47\") " Oct 30 18:36:57.196348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:57.194029 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9a7536b-d288-4ea3-b1d1-f824ec859a47-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9a7536b-d288-4ea3-b1d1-f824ec859a47-openshift-service-ca\") pod \"b9a7536b-d288-4ea3-b1d1-f824ec859a47\" (UID: \"b9a7536b-d288-4ea3-b1d1-f824ec859a47\") " Oct 30 18:36:57.196348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:57.194064 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9a7536b-d288-4ea3-b1d1-f824ec859a47-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9a7536b-d288-4ea3-b1d1-f824ec859a47-maas-api-tls\") pod \"b9a7536b-d288-4ea3-b1d1-f824ec859a47\" (UID: \"b9a7536b-d288-4ea3-b1d1-f824ec859a47\") " Oct 30 18:36:57.196348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:57.194444 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9a7536b-d288-4ea3-b1d1-f824ec859a47-kube-api-access-zh8zx\" (UniqueName: \"kubernetes.io/projected/b9a7536b-d288-4ea3-b1d1-f824ec859a47-kube-api-access-zh8zx\") pod \"b9a7536b-d288-4ea3-b1d1-f824ec859a47\" (UID: \"b9a7536b-d288-4ea3-b1d1-f824ec859a47\") " Oct 30 18:36:57.229347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:57.212218 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9a7536b-d288-4ea3-b1d1-f824ec859a47-openshift-service-ca" pod "b9a7536b-d288-4ea3-b1d1-f824ec859a47" (UID: "b9a7536b-d288-4ea3-b1d1-f824ec859a47"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:57.229347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:57.215921 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a7536b-d288-4ea3-b1d1-f824ec859a47-kube-api-access-zh8zx" pod "b9a7536b-d288-4ea3-b1d1-f824ec859a47" (UID: "b9a7536b-d288-4ea3-b1d1-f824ec859a47"). InnerVolumeSpecName "kube-api-access-zh8zx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:57.229347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:57.217407 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9a7536b-d288-4ea3-b1d1-f824ec859a47-metrics-certs" pod "b9a7536b-d288-4ea3-b1d1-f824ec859a47" (UID: "b9a7536b-d288-4ea3-b1d1-f824ec859a47"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:36:57.229347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:57.228976 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9a7536b-d288-4ea3-b1d1-f824ec859a47-maas-api-tls" pod "b9a7536b-d288-4ea3-b1d1-f824ec859a47" (UID: "b9a7536b-d288-4ea3-b1d1-f824ec859a47"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:36:57.309636 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:57.307820 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9a7536b-d288-4ea3-b1d1-f824ec859a47-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:36:57.309636 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:57.307848 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9a7536b-d288-4ea3-b1d1-f824ec859a47-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:36:57.309636 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:57.307857 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9a7536b-d288-4ea3-b1d1-f824ec859a47-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:36:57.309636 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:57.307866 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh8zx\" (UniqueName: \"kubernetes.io/projected/b9a7536b-d288-4ea3-b1d1-f824ec859a47-kube-api-access-zh8zx\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:36:57.318296 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:57.317019 4455 generic.go:356] "Generic (PLEG): container finished" podID="b9a7536b-d288-4ea3-b1d1-f824ec859a47" containerID="c15d1d7108bd1881634d5fd0f048e1ff9131f51d65cdd24e710785d22c9813f3" exitCode=0 Oct 30 18:36:57.318296 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:57.317087 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m" event={"ID":"b9a7536b-d288-4ea3-b1d1-f824ec859a47","Type":"ContainerDied","Data":"c15d1d7108bd1881634d5fd0f048e1ff9131f51d65cdd24e710785d22c9813f3"} Oct 30 18:36:57.318296 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:57.317141 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m" event={"ID":"b9a7536b-d288-4ea3-b1d1-f824ec859a47","Type":"ContainerDied","Data":"a9cd86cd260ad9a9b917b30f9213dfac7c5a4cf9cce71a11bda58b9882ccc807"} Oct 30 18:36:57.318296 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:57.317168 4455 scope.go:122] "RemoveContainer" containerID="c15d1d7108bd1881634d5fd0f048e1ff9131f51d65cdd24e710785d22c9813f3" Oct 30 18:36:57.318667 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:57.318386 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m" Oct 30 18:36:57.345892 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:57.345628 4455 scope.go:122] "RemoveContainer" containerID="c15d1d7108bd1881634d5fd0f048e1ff9131f51d65cdd24e710785d22c9813f3" Oct 30 18:36:57.346085 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:36:57.346059 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c15d1d7108bd1881634d5fd0f048e1ff9131f51d65cdd24e710785d22c9813f3\": container with ID starting with c15d1d7108bd1881634d5fd0f048e1ff9131f51d65cdd24e710785d22c9813f3 not found: ID does not exist" containerID="c15d1d7108bd1881634d5fd0f048e1ff9131f51d65cdd24e710785d22c9813f3" Oct 30 18:36:57.346215 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:57.346090 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c15d1d7108bd1881634d5fd0f048e1ff9131f51d65cdd24e710785d22c9813f3"} err="failed to get container status \"c15d1d7108bd1881634d5fd0f048e1ff9131f51d65cdd24e710785d22c9813f3\": rpc error: code = NotFound desc = could not find container \"c15d1d7108bd1881634d5fd0f048e1ff9131f51d65cdd24e710785d22c9813f3\": container with ID starting with c15d1d7108bd1881634d5fd0f048e1ff9131f51d65cdd24e710785d22c9813f3 not found: ID does not exist" Oct 30 18:36:57.375045 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:57.375015 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m"] Oct 30 18:36:57.380182 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:57.380155 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5cb3d5-79f9f65d74-bk88m"] Oct 30 18:36:57.740398 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:36:57.734766 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9a7536b-d288-4ea3-b1d1-f824ec859a47" path="/var/lib/kubelet/pods/b9a7536b-d288-4ea3-b1d1-f824ec859a47/volumes" Oct 30 18:37:00.414302 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:37:00.412267 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:37:00.420380 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:00.420352 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m"] Oct 30 18:37:00.420575 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:00.420557 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" podUID="72dfe139-2b2a-4787-b415-6bee9768d82c" containerName="istio-proxy" containerID="cri-o://f8eedc22da04b9d828b7f3dd35edb3b15be631c6ec6857a2a22b9fca07898a45" gracePeriod=30 Oct 30 18:37:01.864245 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:01.864200 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" podUID="38cd4356-6595-419d-a6f2-251106d67bc6" containerName="main" containerID="cri-o://7797cd49a441687ed491ee5b4a78b93fdac09dc81754b09dcc54c6b3c0da1ac7" gracePeriod=45 Oct 30 18:37:01.898409 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:01.898374 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" podUID="38cd4356-6595-419d-a6f2-251106d67bc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.204:8000/ready\": dial tcp 10.134.0.204:8000: connect: connection refused" Oct 30 18:37:02.157916 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.157883 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-a1a31f-6dcdff67b7-gl4jg"] Oct 30 18:37:02.159040 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.159016 4455 cpu_manager.go:414] "containerMap: removing container" podUID="b9a7536b-d288-4ea3-b1d1-f824ec859a47" containerName="maas-api" Oct 30 18:37:02.159040 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.159036 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9a7536b-d288-4ea3-b1d1-f824ec859a47" containerName="maas-api" Oct 30 18:37:02.159299 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.159283 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9a7536b-d288-4ea3-b1d1-f824ec859a47" containerName="maas-api" Oct 30 18:37:02.163464 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.163440 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" Oct 30 18:37:02.164429 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.164408 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a1a31f-6dcdff67b7-gl4jg" Oct 30 18:37:02.189552 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.189519 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-a1a31f\"" Oct 30 18:37:02.197760 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.197726 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-a1a31f-6dcdff67b7-gl4jg"] Oct 30 18:37:02.220566 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.220521 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-a1a31f-dockercfg-kkdgk\"" Oct 30 18:37:02.311375 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.273924 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38cd4356-6595-419d-a6f2-251106d67bc6-kube-api-access-zqklb\" (UniqueName: \"kubernetes.io/projected/38cd4356-6595-419d-a6f2-251106d67bc6-kube-api-access-zqklb\") pod \"38cd4356-6595-419d-a6f2-251106d67bc6\" (UID: \"38cd4356-6595-419d-a6f2-251106d67bc6\") " Oct 30 18:37:02.311375 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.274048 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38cd4356-6595-419d-a6f2-251106d67bc6-home\" (UniqueName: \"kubernetes.io/empty-dir/38cd4356-6595-419d-a6f2-251106d67bc6-home\") pod \"38cd4356-6595-419d-a6f2-251106d67bc6\" (UID: \"38cd4356-6595-419d-a6f2-251106d67bc6\") " Oct 30 18:37:02.311375 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.274108 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38cd4356-6595-419d-a6f2-251106d67bc6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38cd4356-6595-419d-a6f2-251106d67bc6-model-cache\") pod \"38cd4356-6595-419d-a6f2-251106d67bc6\" (UID: \"38cd4356-6595-419d-a6f2-251106d67bc6\") " Oct 30 18:37:02.311375 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.274181 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38cd4356-6595-419d-a6f2-251106d67bc6-tls-certs\" (UniqueName: \"kubernetes.io/secret/38cd4356-6595-419d-a6f2-251106d67bc6-tls-certs\") pod \"38cd4356-6595-419d-a6f2-251106d67bc6\" (UID: \"38cd4356-6595-419d-a6f2-251106d67bc6\") " Oct 30 18:37:02.311375 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.274239 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38cd4356-6595-419d-a6f2-251106d67bc6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/38cd4356-6595-419d-a6f2-251106d67bc6-dshm\") pod \"38cd4356-6595-419d-a6f2-251106d67bc6\" (UID: \"38cd4356-6595-419d-a6f2-251106d67bc6\") " Oct 30 18:37:02.311375 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.283582 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38cd4356-6595-419d-a6f2-251106d67bc6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38cd4356-6595-419d-a6f2-251106d67bc6-tmp-dir\") pod \"38cd4356-6595-419d-a6f2-251106d67bc6\" (UID: \"38cd4356-6595-419d-a6f2-251106d67bc6\") " Oct 30 18:37:02.311375 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.284111 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae8cd58-07cf-40ae-9421-d2c91aee0379-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-a1a31f-6dcdff67b7-gl4jg\" (UID: \"1ae8cd58-07cf-40ae-9421-d2c91aee0379\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-a1a31f-6dcdff67b7-gl4jg" Oct 30 18:37:02.311375 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.284215 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dcx6\" (UniqueName: \"kubernetes.io/projected/1ae8cd58-07cf-40ae-9421-d2c91aee0379-kube-api-access-4dcx6\") pod \"payload-pre-processing-e2e-explicit-a1a31f-6dcdff67b7-gl4jg\" (UID: \"1ae8cd58-07cf-40ae-9421-d2c91aee0379\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-a1a31f-6dcdff67b7-gl4jg" Oct 30 18:37:02.346816 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.319436 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38cd4356-6595-419d-a6f2-251106d67bc6-model-cache" pod "38cd4356-6595-419d-a6f2-251106d67bc6" (UID: "38cd4356-6595-419d-a6f2-251106d67bc6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:02.346816 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.325896 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38cd4356-6595-419d-a6f2-251106d67bc6-kube-api-access-zqklb" pod "38cd4356-6595-419d-a6f2-251106d67bc6" (UID: "38cd4356-6595-419d-a6f2-251106d67bc6"). InnerVolumeSpecName "kube-api-access-zqklb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:02.346816 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.326423 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38cd4356-6595-419d-a6f2-251106d67bc6-home" pod "38cd4356-6595-419d-a6f2-251106d67bc6" (UID: "38cd4356-6595-419d-a6f2-251106d67bc6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:02.346816 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.338983 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38cd4356-6595-419d-a6f2-251106d67bc6-tmp-dir" pod "38cd4356-6595-419d-a6f2-251106d67bc6" (UID: "38cd4356-6595-419d-a6f2-251106d67bc6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:02.499462 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.437989 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae8cd58-07cf-40ae-9421-d2c91aee0379-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-a1a31f-6dcdff67b7-gl4jg\" (UID: \"1ae8cd58-07cf-40ae-9421-d2c91aee0379\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-a1a31f-6dcdff67b7-gl4jg" Oct 30 18:37:02.499462 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.438104 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dcx6\" (UniqueName: \"kubernetes.io/projected/1ae8cd58-07cf-40ae-9421-d2c91aee0379-kube-api-access-4dcx6\") pod \"payload-pre-processing-e2e-explicit-a1a31f-6dcdff67b7-gl4jg\" (UID: \"1ae8cd58-07cf-40ae-9421-d2c91aee0379\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-a1a31f-6dcdff67b7-gl4jg" Oct 30 18:37:02.499462 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.438528 4455 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38cd4356-6595-419d-a6f2-251106d67bc6-home\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:02.499462 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.438547 4455 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38cd4356-6595-419d-a6f2-251106d67bc6-model-cache\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:02.499462 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.438562 4455 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38cd4356-6595-419d-a6f2-251106d67bc6-tmp-dir\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:02.499462 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.438575 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqklb\" (UniqueName: \"kubernetes.io/projected/38cd4356-6595-419d-a6f2-251106d67bc6-kube-api-access-zqklb\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:02.499462 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.439772 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae8cd58-07cf-40ae-9421-d2c91aee0379-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-a1a31f-6dcdff67b7-gl4jg\" (UID: \"1ae8cd58-07cf-40ae-9421-d2c91aee0379\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-a1a31f-6dcdff67b7-gl4jg" Oct 30 18:37:02.577562 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.577095 4455 generic.go:356] "Generic (PLEG): container finished" podID="38cd4356-6595-419d-a6f2-251106d67bc6" containerID="7797cd49a441687ed491ee5b4a78b93fdac09dc81754b09dcc54c6b3c0da1ac7" exitCode=0 Oct 30 18:37:02.577562 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.577399 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" event={"ID":"38cd4356-6595-419d-a6f2-251106d67bc6","Type":"ContainerDied","Data":"7797cd49a441687ed491ee5b4a78b93fdac09dc81754b09dcc54c6b3c0da1ac7"} Oct 30 18:37:02.577562 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.577440 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" event={"ID":"38cd4356-6595-419d-a6f2-251106d67bc6","Type":"ContainerDied","Data":"a2aaba0cebb90e526c8a938c6b1019f8500c6d8014691f73786bcb6aeef32d9a"} Oct 30 18:37:02.577562 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.577462 4455 scope.go:122] "RemoveContainer" containerID="7797cd49a441687ed491ee5b4a78b93fdac09dc81754b09dcc54c6b3c0da1ac7" Oct 30 18:37:02.577562 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.577468 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq" Oct 30 18:37:02.586455 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.586401 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a1a31f-5858ff7c48-xz2tp"] Oct 30 18:37:02.590261 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.586847 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38cd4356-6595-419d-a6f2-251106d67bc6-tls-certs" pod "38cd4356-6595-419d-a6f2-251106d67bc6" (UID: "38cd4356-6595-419d-a6f2-251106d67bc6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:02.590261 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.587867 4455 cpu_manager.go:414] "containerMap: removing container" podUID="38cd4356-6595-419d-a6f2-251106d67bc6" containerName="main" Oct 30 18:37:02.590261 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.587889 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38cd4356-6595-419d-a6f2-251106d67bc6" containerName="main" Oct 30 18:37:02.590261 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.588118 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="38cd4356-6595-419d-a6f2-251106d67bc6" containerName="main" Oct 30 18:37:02.608846 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.607403 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38cd4356-6595-419d-a6f2-251106d67bc6-dshm" pod "38cd4356-6595-419d-a6f2-251106d67bc6" (UID: "38cd4356-6595-419d-a6f2-251106d67bc6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:02.612403 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.610002 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dcx6\" (UniqueName: \"kubernetes.io/projected/1ae8cd58-07cf-40ae-9421-d2c91aee0379-kube-api-access-4dcx6\") pod \"payload-pre-processing-e2e-explicit-a1a31f-6dcdff67b7-gl4jg\" (UID: \"1ae8cd58-07cf-40ae-9421-d2c91aee0379\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-a1a31f-6dcdff67b7-gl4jg" Oct 30 18:37:02.634860 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.634790 4455 scope.go:122] "RemoveContainer" containerID="7797cd49a441687ed491ee5b4a78b93fdac09dc81754b09dcc54c6b3c0da1ac7" Oct 30 18:37:02.635292 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:37:02.635261 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7797cd49a441687ed491ee5b4a78b93fdac09dc81754b09dcc54c6b3c0da1ac7\": container with ID starting with 7797cd49a441687ed491ee5b4a78b93fdac09dc81754b09dcc54c6b3c0da1ac7 not found: ID does not exist" containerID="7797cd49a441687ed491ee5b4a78b93fdac09dc81754b09dcc54c6b3c0da1ac7" Oct 30 18:37:02.635797 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.635303 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7797cd49a441687ed491ee5b4a78b93fdac09dc81754b09dcc54c6b3c0da1ac7"} err="failed to get container status \"7797cd49a441687ed491ee5b4a78b93fdac09dc81754b09dcc54c6b3c0da1ac7\": rpc error: code = NotFound desc = could not find container \"7797cd49a441687ed491ee5b4a78b93fdac09dc81754b09dcc54c6b3c0da1ac7\": container with ID starting with 7797cd49a441687ed491ee5b4a78b93fdac09dc81754b09dcc54c6b3c0da1ac7 not found: ID does not exist" Oct 30 18:37:02.635797 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.635509 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-a1a31f-5858ff7c48-xz2tp" Oct 30 18:37:02.636538 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.636513 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a1a31f-5858ff7c48-xz2tp"] Oct 30 18:37:02.664850 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.664822 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k62vt\" (UniqueName: \"kubernetes.io/projected/502021d0-cc42-404e-a0f7-67f7392c82bf-kube-api-access-k62vt\") pod \"payload-processing-e2e-explicit-a1a31f-5858ff7c48-xz2tp\" (UID: \"502021d0-cc42-404e-a0f7-67f7392c82bf\") " pod="openshift-ingress/payload-processing-e2e-explicit-a1a31f-5858ff7c48-xz2tp" Oct 30 18:37:02.665029 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.664978 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/502021d0-cc42-404e-a0f7-67f7392c82bf-plugins-config-volume\") pod \"payload-processing-e2e-explicit-a1a31f-5858ff7c48-xz2tp\" (UID: \"502021d0-cc42-404e-a0f7-67f7392c82bf\") " pod="openshift-ingress/payload-processing-e2e-explicit-a1a31f-5858ff7c48-xz2tp" Oct 30 18:37:02.665217 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.665198 4455 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38cd4356-6595-419d-a6f2-251106d67bc6-tls-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:02.665263 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.665220 4455 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38cd4356-6595-419d-a6f2-251106d67bc6-dshm\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:02.766274 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.766167 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k62vt\" (UniqueName: \"kubernetes.io/projected/502021d0-cc42-404e-a0f7-67f7392c82bf-kube-api-access-k62vt\") pod \"payload-processing-e2e-explicit-a1a31f-5858ff7c48-xz2tp\" (UID: \"502021d0-cc42-404e-a0f7-67f7392c82bf\") " pod="openshift-ingress/payload-processing-e2e-explicit-a1a31f-5858ff7c48-xz2tp" Oct 30 18:37:02.766274 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.766272 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/502021d0-cc42-404e-a0f7-67f7392c82bf-plugins-config-volume\") pod \"payload-processing-e2e-explicit-a1a31f-5858ff7c48-xz2tp\" (UID: \"502021d0-cc42-404e-a0f7-67f7392c82bf\") " pod="openshift-ingress/payload-processing-e2e-explicit-a1a31f-5858ff7c48-xz2tp" Oct 30 18:37:02.767047 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.767025 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/502021d0-cc42-404e-a0f7-67f7392c82bf-plugins-config-volume\") pod \"payload-processing-e2e-explicit-a1a31f-5858ff7c48-xz2tp\" (UID: \"502021d0-cc42-404e-a0f7-67f7392c82bf\") " pod="openshift-ingress/payload-processing-e2e-explicit-a1a31f-5858ff7c48-xz2tp" Oct 30 18:37:02.784974 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.783409 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a1a31f-6dcdff67b7-gl4jg" Oct 30 18:37:02.793979 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.793951 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k62vt\" (UniqueName: \"kubernetes.io/projected/502021d0-cc42-404e-a0f7-67f7392c82bf-kube-api-access-k62vt\") pod \"payload-processing-e2e-explicit-a1a31f-5858ff7c48-xz2tp\" (UID: \"502021d0-cc42-404e-a0f7-67f7392c82bf\") " pod="openshift-ingress/payload-processing-e2e-explicit-a1a31f-5858ff7c48-xz2tp" Oct 30 18:37:02.944205 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.944149 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq"] Oct 30 18:37:02.950123 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.950097 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-3b8bf3/auth-test-model-3b8bf3-kserve-74447bd545-lkxtq"] Oct 30 18:37:02.950605 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:02.950535 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-a1a31f-5858ff7c48-xz2tp" Oct 30 18:37:03.113194 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:03.113169 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-a1a31f-6dcdff67b7-gl4jg"] Oct 30 18:37:03.284388 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:03.284354 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a1a31f-5858ff7c48-xz2tp"] Oct 30 18:37:03.285080 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:37:03.285052 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod502021d0_cc42_404e_a0f7_67f7392c82bf.slice/crio-046527b8a3238903bfbf7916aed75e6fdbe9d603115940e3db50cbb82f6b5685 WatchSource:0}: Error finding container 046527b8a3238903bfbf7916aed75e6fdbe9d603115940e3db50cbb82f6b5685: Status 404 returned error can't find the container with id 046527b8a3238903bfbf7916aed75e6fdbe9d603115940e3db50cbb82f6b5685 Oct 30 18:37:03.589672 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:03.589556 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-a1a31f-5858ff7c48-xz2tp" event={"ID":"502021d0-cc42-404e-a0f7-67f7392c82bf","Type":"ContainerStarted","Data":"f75b149c6027c34cb527b3d8ada9717dc241c6b7aa391361cd2c4cb1770a090a"} Oct 30 18:37:03.589672 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:03.589622 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-a1a31f-5858ff7c48-xz2tp" event={"ID":"502021d0-cc42-404e-a0f7-67f7392c82bf","Type":"ContainerStarted","Data":"046527b8a3238903bfbf7916aed75e6fdbe9d603115940e3db50cbb82f6b5685"} Oct 30 18:37:03.590447 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:03.590423 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-a1a31f-5858ff7c48-xz2tp" Oct 30 18:37:03.593970 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:03.593937 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a1a31f-6dcdff67b7-gl4jg" event={"ID":"1ae8cd58-07cf-40ae-9421-d2c91aee0379","Type":"ContainerStarted","Data":"8b51075b6d9ea930c3b2349c2d9a8aadef5a8a9c1674ac7a822914ef62502b8f"} Oct 30 18:37:03.594126 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:03.593991 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a1a31f-6dcdff67b7-gl4jg" event={"ID":"1ae8cd58-07cf-40ae-9421-d2c91aee0379","Type":"ContainerStarted","Data":"c39dcdcb73103cf812ab6883ee67f80f3ab88004f053385cf741eb41d7d94f50"} Oct 30 18:37:03.594766 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:03.594739 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a1a31f-6dcdff67b7-gl4jg" Oct 30 18:37:03.636857 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:03.636824 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38cd4356-6595-419d-a6f2-251106d67bc6" path="/var/lib/kubelet/pods/38cd4356-6595-419d-a6f2-251106d67bc6/volumes" Oct 30 18:37:03.642046 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:03.642002 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-a1a31f-5858ff7c48-xz2tp" podStartSLOduration=1.641989754 podStartE2EDuration="1.641989754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:03.638743075 +0000 UTC m=+2676.126380109" watchObservedRunningTime="2026-10-30 18:37:03.641989754 +0000 UTC m=+2676.129626788" Oct 30 18:37:03.681711 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:03.681655 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a1a31f-6dcdff67b7-gl4jg" podStartSLOduration=1.681638012 podStartE2EDuration="1.681638012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:03.681003216 +0000 UTC m=+2676.168640250" watchObservedRunningTime="2026-10-30 18:37:03.681638012 +0000 UTC m=+2676.169275048" Oct 30 18:37:05.629785 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:05.629751 4455 generic.go:356] "Generic (PLEG): container finished" podID="72dfe139-2b2a-4787-b415-6bee9768d82c" containerID="f8eedc22da04b9d828b7f3dd35edb3b15be631c6ec6857a2a22b9fca07898a45" exitCode=0 Oct 30 18:37:05.642285 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:05.642227 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" event={"ID":"72dfe139-2b2a-4787-b415-6bee9768d82c","Type":"ContainerDied","Data":"f8eedc22da04b9d828b7f3dd35edb3b15be631c6ec6857a2a22b9fca07898a45"} Oct 30 18:37:05.850058 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:05.850028 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" Oct 30 18:37:06.040648 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:06.040565 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72dfe139-2b2a-4787-b415-6bee9768d82c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72dfe139-2b2a-4787-b415-6bee9768d82c-istiod-ca-cert\") pod \"72dfe139-2b2a-4787-b415-6bee9768d82c\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " Oct 30 18:37:06.040831 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:06.040707 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72dfe139-2b2a-4787-b415-6bee9768d82c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72dfe139-2b2a-4787-b415-6bee9768d82c-istio-envoy\") pod \"72dfe139-2b2a-4787-b415-6bee9768d82c\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " Oct 30 18:37:06.040831 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:06.040806 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72dfe139-2b2a-4787-b415-6bee9768d82c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72dfe139-2b2a-4787-b415-6bee9768d82c-istio-podinfo\") pod \"72dfe139-2b2a-4787-b415-6bee9768d82c\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " Oct 30 18:37:06.040920 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:06.040846 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72dfe139-2b2a-4787-b415-6bee9768d82c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72dfe139-2b2a-4787-b415-6bee9768d82c-istio-data\") pod \"72dfe139-2b2a-4787-b415-6bee9768d82c\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " Oct 30 18:37:06.040920 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:06.040886 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72dfe139-2b2a-4787-b415-6bee9768d82c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72dfe139-2b2a-4787-b415-6bee9768d82c-workload-certs\") pod \"72dfe139-2b2a-4787-b415-6bee9768d82c\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " Oct 30 18:37:06.041008 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:06.040921 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72dfe139-2b2a-4787-b415-6bee9768d82c-istiod-ca-cert" pod "72dfe139-2b2a-4787-b415-6bee9768d82c" (UID: "72dfe139-2b2a-4787-b415-6bee9768d82c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:06.041008 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:06.040973 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72dfe139-2b2a-4787-b415-6bee9768d82c-kube-api-access-t6kx2\" (UniqueName: \"kubernetes.io/projected/72dfe139-2b2a-4787-b415-6bee9768d82c-kube-api-access-t6kx2\") pod \"72dfe139-2b2a-4787-b415-6bee9768d82c\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " Oct 30 18:37:06.041079 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:06.041041 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72dfe139-2b2a-4787-b415-6bee9768d82c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72dfe139-2b2a-4787-b415-6bee9768d82c-workload-socket\") pod \"72dfe139-2b2a-4787-b415-6bee9768d82c\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " Oct 30 18:37:06.041135 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:06.041101 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72dfe139-2b2a-4787-b415-6bee9768d82c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72dfe139-2b2a-4787-b415-6bee9768d82c-credential-socket\") pod \"72dfe139-2b2a-4787-b415-6bee9768d82c\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " Oct 30 18:37:06.041195 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:06.041138 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72dfe139-2b2a-4787-b415-6bee9768d82c-istio-token\" (UniqueName: \"kubernetes.io/projected/72dfe139-2b2a-4787-b415-6bee9768d82c-istio-token\") pod \"72dfe139-2b2a-4787-b415-6bee9768d82c\" (UID: \"72dfe139-2b2a-4787-b415-6bee9768d82c\") " Oct 30 18:37:06.041195 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:06.041169 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72dfe139-2b2a-4787-b415-6bee9768d82c-istio-data" pod "72dfe139-2b2a-4787-b415-6bee9768d82c" (UID: "72dfe139-2b2a-4787-b415-6bee9768d82c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:06.041317 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:06.041187 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72dfe139-2b2a-4787-b415-6bee9768d82c-workload-certs" pod "72dfe139-2b2a-4787-b415-6bee9768d82c" (UID: "72dfe139-2b2a-4787-b415-6bee9768d82c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:06.041467 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:06.041443 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72dfe139-2b2a-4787-b415-6bee9768d82c-workload-socket" pod "72dfe139-2b2a-4787-b415-6bee9768d82c" (UID: "72dfe139-2b2a-4787-b415-6bee9768d82c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:06.041536 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:06.041461 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72dfe139-2b2a-4787-b415-6bee9768d82c-credential-socket" pod "72dfe139-2b2a-4787-b415-6bee9768d82c" (UID: "72dfe139-2b2a-4787-b415-6bee9768d82c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:06.041945 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:06.041877 4455 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72dfe139-2b2a-4787-b415-6bee9768d82c-istio-data\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:06.041945 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:06.041900 4455 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72dfe139-2b2a-4787-b415-6bee9768d82c-workload-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:06.041945 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:06.041910 4455 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72dfe139-2b2a-4787-b415-6bee9768d82c-workload-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:06.041945 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:06.041921 4455 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72dfe139-2b2a-4787-b415-6bee9768d82c-credential-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:06.041945 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:06.041937 4455 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72dfe139-2b2a-4787-b415-6bee9768d82c-istiod-ca-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:06.043503 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:06.043477 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72dfe139-2b2a-4787-b415-6bee9768d82c-istio-token" pod "72dfe139-2b2a-4787-b415-6bee9768d82c" (UID: "72dfe139-2b2a-4787-b415-6bee9768d82c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:06.043619 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:06.043509 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72dfe139-2b2a-4787-b415-6bee9768d82c-istio-envoy" pod "72dfe139-2b2a-4787-b415-6bee9768d82c" (UID: "72dfe139-2b2a-4787-b415-6bee9768d82c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:06.043619 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:06.043548 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72dfe139-2b2a-4787-b415-6bee9768d82c-istio-podinfo" pod "72dfe139-2b2a-4787-b415-6bee9768d82c" (UID: "72dfe139-2b2a-4787-b415-6bee9768d82c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:37:06.043754 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:06.043731 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72dfe139-2b2a-4787-b415-6bee9768d82c-kube-api-access-t6kx2" pod "72dfe139-2b2a-4787-b415-6bee9768d82c" (UID: "72dfe139-2b2a-4787-b415-6bee9768d82c"). InnerVolumeSpecName "kube-api-access-t6kx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:06.143907 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:06.143872 4455 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72dfe139-2b2a-4787-b415-6bee9768d82c-istio-envoy\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:06.143907 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:06.143902 4455 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72dfe139-2b2a-4787-b415-6bee9768d82c-istio-podinfo\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:06.144126 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:06.143915 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6kx2\" (UniqueName: \"kubernetes.io/projected/72dfe139-2b2a-4787-b415-6bee9768d82c-kube-api-access-t6kx2\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:06.144126 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:06.143930 4455 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72dfe139-2b2a-4787-b415-6bee9768d82c-istio-token\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:06.638934 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:06.638907 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" Oct 30 18:37:06.639427 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:06.638906 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m" event={"ID":"72dfe139-2b2a-4787-b415-6bee9768d82c","Type":"ContainerDied","Data":"621e57bd13d5cd50c6162c3e4272be7822fdea77b149856094cecfab800a820c"} Oct 30 18:37:06.639427 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:06.639034 4455 scope.go:122] "RemoveContainer" containerID="f8eedc22da04b9d828b7f3dd35edb3b15be631c6ec6857a2a22b9fca07898a45" Oct 30 18:37:06.706781 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:06.706748 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m"] Oct 30 18:37:06.715133 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:06.715106 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-spd6m"] Oct 30 18:37:07.635872 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:07.635842 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72dfe139-2b2a-4787-b415-6bee9768d82c" path="/var/lib/kubelet/pods/72dfe139-2b2a-4787-b415-6bee9768d82c/volumes" Oct 30 18:37:08.330750 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:08.330723 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-58d08b-866bd6bfff-8vjgc" Oct 30 18:37:08.331191 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:08.330803 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-58d08b-c58b968db-9f7zq" Oct 30 18:37:09.450957 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:09.450923 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs"] Oct 30 18:37:09.451992 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:09.451968 4455 cpu_manager.go:414] "containerMap: removing container" podUID="72dfe139-2b2a-4787-b415-6bee9768d82c" containerName="istio-proxy" Oct 30 18:37:09.451992 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:09.451994 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72dfe139-2b2a-4787-b415-6bee9768d82c" containerName="istio-proxy" Oct 30 18:37:09.452259 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:09.452241 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="72dfe139-2b2a-4787-b415-6bee9768d82c" containerName="istio-proxy" Oct 30 18:37:09.456431 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:09.456411 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs" Oct 30 18:37:09.498973 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:09.498940 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs"] Oct 30 18:37:09.514559 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:09.514522 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96e42878-31cc-4d86-b428-e39a8aa72218-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z2svs\" (UID: \"96e42878-31cc-4d86-b428-e39a8aa72218\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs" Oct 30 18:37:09.514763 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:09.514587 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/96e42878-31cc-4d86-b428-e39a8aa72218-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z2svs\" (UID: \"96e42878-31cc-4d86-b428-e39a8aa72218\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs" Oct 30 18:37:09.514763 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:09.514629 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/96e42878-31cc-4d86-b428-e39a8aa72218-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z2svs\" (UID: \"96e42878-31cc-4d86-b428-e39a8aa72218\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs" Oct 30 18:37:09.514763 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:09.514658 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/96e42878-31cc-4d86-b428-e39a8aa72218-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z2svs\" (UID: \"96e42878-31cc-4d86-b428-e39a8aa72218\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs" Oct 30 18:37:09.514938 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:09.514832 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/96e42878-31cc-4d86-b428-e39a8aa72218-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z2svs\" (UID: \"96e42878-31cc-4d86-b428-e39a8aa72218\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs" Oct 30 18:37:09.514996 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:09.514953 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jm7w\" (UniqueName: \"kubernetes.io/projected/96e42878-31cc-4d86-b428-e39a8aa72218-kube-api-access-6jm7w\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z2svs\" (UID: \"96e42878-31cc-4d86-b428-e39a8aa72218\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs" Oct 30 18:37:09.515053 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:09.515012 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/96e42878-31cc-4d86-b428-e39a8aa72218-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z2svs\" (UID: \"96e42878-31cc-4d86-b428-e39a8aa72218\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs" Oct 30 18:37:09.616688 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:09.616655 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96e42878-31cc-4d86-b428-e39a8aa72218-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z2svs\" (UID: \"96e42878-31cc-4d86-b428-e39a8aa72218\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs" Oct 30 18:37:09.616688 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:09.616701 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/96e42878-31cc-4d86-b428-e39a8aa72218-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z2svs\" (UID: \"96e42878-31cc-4d86-b428-e39a8aa72218\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs" Oct 30 18:37:09.616962 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:09.616885 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/96e42878-31cc-4d86-b428-e39a8aa72218-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z2svs\" (UID: \"96e42878-31cc-4d86-b428-e39a8aa72218\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs" Oct 30 18:37:09.616962 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:09.616932 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/96e42878-31cc-4d86-b428-e39a8aa72218-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z2svs\" (UID: \"96e42878-31cc-4d86-b428-e39a8aa72218\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs" Oct 30 18:37:09.617124 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:09.617106 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/96e42878-31cc-4d86-b428-e39a8aa72218-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z2svs\" (UID: \"96e42878-31cc-4d86-b428-e39a8aa72218\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs" Oct 30 18:37:09.617247 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:09.617229 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jm7w\" (UniqueName: \"kubernetes.io/projected/96e42878-31cc-4d86-b428-e39a8aa72218-kube-api-access-6jm7w\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z2svs\" (UID: \"96e42878-31cc-4d86-b428-e39a8aa72218\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs" Oct 30 18:37:09.617369 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:09.617299 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/96e42878-31cc-4d86-b428-e39a8aa72218-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z2svs\" (UID: \"96e42878-31cc-4d86-b428-e39a8aa72218\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs" Oct 30 18:37:09.618071 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:09.618045 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/96e42878-31cc-4d86-b428-e39a8aa72218-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z2svs\" (UID: \"96e42878-31cc-4d86-b428-e39a8aa72218\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs" Oct 30 18:37:09.619372 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:09.619349 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/96e42878-31cc-4d86-b428-e39a8aa72218-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z2svs\" (UID: \"96e42878-31cc-4d86-b428-e39a8aa72218\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs" Oct 30 18:37:09.619714 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:09.619684 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/96e42878-31cc-4d86-b428-e39a8aa72218-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z2svs\" (UID: \"96e42878-31cc-4d86-b428-e39a8aa72218\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs" Oct 30 18:37:09.619947 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:09.619926 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/96e42878-31cc-4d86-b428-e39a8aa72218-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z2svs\" (UID: \"96e42878-31cc-4d86-b428-e39a8aa72218\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs" Oct 30 18:37:09.620092 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:09.620074 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/96e42878-31cc-4d86-b428-e39a8aa72218-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z2svs\" (UID: \"96e42878-31cc-4d86-b428-e39a8aa72218\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs" Oct 30 18:37:09.641887 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:09.641860 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96e42878-31cc-4d86-b428-e39a8aa72218-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z2svs\" (UID: \"96e42878-31cc-4d86-b428-e39a8aa72218\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs" Oct 30 18:37:09.642449 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:09.642426 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jm7w\" (UniqueName: \"kubernetes.io/projected/96e42878-31cc-4d86-b428-e39a8aa72218-kube-api-access-6jm7w\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z2svs\" (UID: \"96e42878-31cc-4d86-b428-e39a8aa72218\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs" Oct 30 18:37:09.770302 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:09.770267 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs" Oct 30 18:37:10.120283 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:10.120244 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs"] Oct 30 18:37:10.120787 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:37:10.120763 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e42878_31cc_4d86_b428_e39a8aa72218.slice/crio-3e9f8bc9ec22891fbb476505e1295e5f52568930395db12b7e53882fe56943fa WatchSource:0}: Error finding container 3e9f8bc9ec22891fbb476505e1295e5f52568930395db12b7e53882fe56943fa: Status 404 returned error can't find the container with id 3e9f8bc9ec22891fbb476505e1295e5f52568930395db12b7e53882fe56943fa Oct 30 18:37:10.123562 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:10.123524 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:37:10.123664 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:10.123598 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:37:10.685120 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:10.685044 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs" event={"ID":"96e42878-31cc-4d86-b428-e39a8aa72218","Type":"ContainerStarted","Data":"88ca55b7819e9a1e970b04be6c0d19e087497312ad24e57a9b20dec92345509f"} Oct 30 18:37:10.685120 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:10.685106 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs" event={"ID":"96e42878-31cc-4d86-b428-e39a8aa72218","Type":"ContainerStarted","Data":"3e9f8bc9ec22891fbb476505e1295e5f52568930395db12b7e53882fe56943fa"} Oct 30 18:37:10.685753 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:10.685156 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs" Oct 30 18:37:10.754209 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:10.754148 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs" podStartSLOduration=1.754132136 podStartE2EDuration="1.754132136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:10.740592723 +0000 UTC m=+2683.228229766" watchObservedRunningTime="2026-10-30 18:37:10.754132136 +0000 UTC m=+2683.241769170" Oct 30 18:37:11.738903 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:11.738864 4455 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-z2svs container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:37:11.739426 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:11.738915 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs" podUID="96e42878-31cc-4d86-b428-e39a8aa72218" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:37:14.644347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:14.636644 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-a1a31f-5858ff7c48-xz2tp" Oct 30 18:37:14.650512 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:14.650480 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a1a31f-6dcdff67b7-gl4jg" Oct 30 18:37:14.734793 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:14.734765 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs" Oct 30 18:37:28.853199 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:28.853156 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w"] Oct 30 18:37:28.853727 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:28.853443 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w" podUID="d6058153-6f0e-4d2f-ac5c-53be26e376be" containerName="maas-api" containerID="cri-o://d6102c4863b50757fb6eab37a974772adbb6d3bd59d94da86b94dca2f4d680e9" gracePeriod=30 Oct 30 18:37:29.291350 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:29.290402 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w" Oct 30 18:37:29.402288 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:29.395877 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6058153-6f0e-4d2f-ac5c-53be26e376be-kube-api-access-kqf56\" (UniqueName: \"kubernetes.io/projected/d6058153-6f0e-4d2f-ac5c-53be26e376be-kube-api-access-kqf56\") pod \"d6058153-6f0e-4d2f-ac5c-53be26e376be\" (UID: \"d6058153-6f0e-4d2f-ac5c-53be26e376be\") " Oct 30 18:37:29.402288 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:29.395957 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6058153-6f0e-4d2f-ac5c-53be26e376be-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6058153-6f0e-4d2f-ac5c-53be26e376be-maas-api-tls\") pod \"d6058153-6f0e-4d2f-ac5c-53be26e376be\" (UID: \"d6058153-6f0e-4d2f-ac5c-53be26e376be\") " Oct 30 18:37:29.402288 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:29.396074 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6058153-6f0e-4d2f-ac5c-53be26e376be-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6058153-6f0e-4d2f-ac5c-53be26e376be-openshift-service-ca\") pod \"d6058153-6f0e-4d2f-ac5c-53be26e376be\" (UID: \"d6058153-6f0e-4d2f-ac5c-53be26e376be\") " Oct 30 18:37:29.402288 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:29.396138 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6058153-6f0e-4d2f-ac5c-53be26e376be-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6058153-6f0e-4d2f-ac5c-53be26e376be-metrics-certs\") pod \"d6058153-6f0e-4d2f-ac5c-53be26e376be\" (UID: \"d6058153-6f0e-4d2f-ac5c-53be26e376be\") " Oct 30 18:37:29.402288 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:29.399164 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6058153-6f0e-4d2f-ac5c-53be26e376be-openshift-service-ca" pod "d6058153-6f0e-4d2f-ac5c-53be26e376be" (UID: "d6058153-6f0e-4d2f-ac5c-53be26e376be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:29.402288 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:29.399827 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6058153-6f0e-4d2f-ac5c-53be26e376be-metrics-certs" pod "d6058153-6f0e-4d2f-ac5c-53be26e376be" (UID: "d6058153-6f0e-4d2f-ac5c-53be26e376be"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:29.403003 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:29.402977 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6058153-6f0e-4d2f-ac5c-53be26e376be-maas-api-tls" pod "d6058153-6f0e-4d2f-ac5c-53be26e376be" (UID: "d6058153-6f0e-4d2f-ac5c-53be26e376be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:29.446610 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:29.446547 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6058153-6f0e-4d2f-ac5c-53be26e376be-kube-api-access-kqf56" pod "d6058153-6f0e-4d2f-ac5c-53be26e376be" (UID: "d6058153-6f0e-4d2f-ac5c-53be26e376be"). InnerVolumeSpecName "kube-api-access-kqf56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:29.540346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:29.525187 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqf56\" (UniqueName: \"kubernetes.io/projected/d6058153-6f0e-4d2f-ac5c-53be26e376be-kube-api-access-kqf56\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.540346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:29.525223 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6058153-6f0e-4d2f-ac5c-53be26e376be-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.540346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:29.525238 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6058153-6f0e-4d2f-ac5c-53be26e376be-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.540346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:29.525251 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6058153-6f0e-4d2f-ac5c-53be26e376be-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.953084 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:29.952979 4455 generic.go:356] "Generic (PLEG): container finished" podID="d6058153-6f0e-4d2f-ac5c-53be26e376be" containerID="d6102c4863b50757fb6eab37a974772adbb6d3bd59d94da86b94dca2f4d680e9" exitCode=0 Oct 30 18:37:29.953593 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:29.953111 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w" event={"ID":"d6058153-6f0e-4d2f-ac5c-53be26e376be","Type":"ContainerDied","Data":"d6102c4863b50757fb6eab37a974772adbb6d3bd59d94da86b94dca2f4d680e9"} Oct 30 18:37:29.953593 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:29.953205 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w" event={"ID":"d6058153-6f0e-4d2f-ac5c-53be26e376be","Type":"ContainerDied","Data":"653b29ecbf475babbc722b7cc02891ba28d8f6fa374fceff6b09d6775834128d"} Oct 30 18:37:29.953593 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:29.953248 4455 scope.go:122] "RemoveContainer" containerID="d6102c4863b50757fb6eab37a974772adbb6d3bd59d94da86b94dca2f4d680e9" Oct 30 18:37:29.953593 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:29.953474 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w" Oct 30 18:37:29.973220 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:29.973194 4455 scope.go:122] "RemoveContainer" containerID="d6102c4863b50757fb6eab37a974772adbb6d3bd59d94da86b94dca2f4d680e9" Oct 30 18:37:29.974491 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:37:29.974459 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6102c4863b50757fb6eab37a974772adbb6d3bd59d94da86b94dca2f4d680e9\": container with ID starting with d6102c4863b50757fb6eab37a974772adbb6d3bd59d94da86b94dca2f4d680e9 not found: ID does not exist" containerID="d6102c4863b50757fb6eab37a974772adbb6d3bd59d94da86b94dca2f4d680e9" Oct 30 18:37:29.974582 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:29.974500 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6102c4863b50757fb6eab37a974772adbb6d3bd59d94da86b94dca2f4d680e9"} err="failed to get container status \"d6102c4863b50757fb6eab37a974772adbb6d3bd59d94da86b94dca2f4d680e9\": rpc error: code = NotFound desc = could not find container \"d6102c4863b50757fb6eab37a974772adbb6d3bd59d94da86b94dca2f4d680e9\": container with ID starting with d6102c4863b50757fb6eab37a974772adbb6d3bd59d94da86b94dca2f4d680e9 not found: ID does not exist" Oct 30 18:37:30.063811 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:30.063777 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w"] Oct 30 18:37:30.074483 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:30.074447 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b00d69-7d74dbc67f-5vp5w"] Oct 30 18:37:30.386602 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:37:30.386564 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:37:31.636362 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:31.636254 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6058153-6f0e-4d2f-ac5c-53be26e376be" path="/var/lib/kubelet/pods/d6058153-6f0e-4d2f-ac5c-53be26e376be/volumes" Oct 30 18:37:48.982922 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:48.982875 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6"] Oct 30 18:37:48.983590 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:48.983159 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" podUID="a8a6e230-4b8f-4d8d-af42-d6660522152e" containerName="istio-proxy" containerID="cri-o://e2002cc6bfa68049e67590ef9ea1a59f8c518482ed44f52c2ff7d730d7e55671" gracePeriod=30 Oct 30 18:37:49.019866 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:49.019826 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv"] Oct 30 18:37:49.020108 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:49.020077 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" podUID="b41517e2-cc07-4530-8ec9-092c2a92ad47" containerName="istio-proxy" containerID="cri-o://ce80d0cadfc75aac9e1f53d048259d4126674e32cbe065b2a660e32f08699c89" gracePeriod=30 Oct 30 18:37:49.411436 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:49.411359 4455 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:37:49.411436 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:49.411416 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" podUID="a8a6e230-4b8f-4d8d-af42-d6660522152e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:37:50.207787 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:50.207751 4455 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:37:50.208219 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:50.207829 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" podUID="b41517e2-cc07-4530-8ec9-092c2a92ad47" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:37:54.280033 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.279923 4455 generic.go:356] "Generic (PLEG): container finished" podID="a8a6e230-4b8f-4d8d-af42-d6660522152e" containerID="e2002cc6bfa68049e67590ef9ea1a59f8c518482ed44f52c2ff7d730d7e55671" exitCode=0 Oct 30 18:37:54.280610 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.280117 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" event={"ID":"a8a6e230-4b8f-4d8d-af42-d6660522152e","Type":"ContainerDied","Data":"e2002cc6bfa68049e67590ef9ea1a59f8c518482ed44f52c2ff7d730d7e55671"} Oct 30 18:37:54.285430 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.285397 4455 generic.go:356] "Generic (PLEG): container finished" podID="b41517e2-cc07-4530-8ec9-092c2a92ad47" containerID="ce80d0cadfc75aac9e1f53d048259d4126674e32cbe065b2a660e32f08699c89" exitCode=0 Oct 30 18:37:54.285832 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.285799 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" event={"ID":"b41517e2-cc07-4530-8ec9-092c2a92ad47","Type":"ContainerDied","Data":"ce80d0cadfc75aac9e1f53d048259d4126674e32cbe065b2a660e32f08699c89"} Oct 30 18:37:54.340510 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.340476 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" Oct 30 18:37:54.466651 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.466608 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb"] Oct 30 18:37:54.482409 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.482030 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b41517e2-cc07-4530-8ec9-092c2a92ad47-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41517e2-cc07-4530-8ec9-092c2a92ad47-workload-socket\") pod \"b41517e2-cc07-4530-8ec9-092c2a92ad47\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " Oct 30 18:37:54.482409 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.482112 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b41517e2-cc07-4530-8ec9-092c2a92ad47-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b41517e2-cc07-4530-8ec9-092c2a92ad47-istiod-ca-cert\") pod \"b41517e2-cc07-4530-8ec9-092c2a92ad47\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " Oct 30 18:37:54.482409 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.482232 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b41517e2-cc07-4530-8ec9-092c2a92ad47-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b41517e2-cc07-4530-8ec9-092c2a92ad47-istio-envoy\") pod \"b41517e2-cc07-4530-8ec9-092c2a92ad47\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " Oct 30 18:37:54.483917 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.482358 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b41517e2-cc07-4530-8ec9-092c2a92ad47-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b41517e2-cc07-4530-8ec9-092c2a92ad47-istio-podinfo\") pod \"b41517e2-cc07-4530-8ec9-092c2a92ad47\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " Oct 30 18:37:54.483917 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.482840 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b41517e2-cc07-4530-8ec9-092c2a92ad47-istiod-ca-cert" pod "b41517e2-cc07-4530-8ec9-092c2a92ad47" (UID: "b41517e2-cc07-4530-8ec9-092c2a92ad47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:54.483917 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.482927 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b41517e2-cc07-4530-8ec9-092c2a92ad47-kube-api-access-nqfzv\" (UniqueName: \"kubernetes.io/projected/b41517e2-cc07-4530-8ec9-092c2a92ad47-kube-api-access-nqfzv\") pod \"b41517e2-cc07-4530-8ec9-092c2a92ad47\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " Oct 30 18:37:54.483917 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.483043 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b41517e2-cc07-4530-8ec9-092c2a92ad47-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b41517e2-cc07-4530-8ec9-092c2a92ad47-workload-certs\") pod \"b41517e2-cc07-4530-8ec9-092c2a92ad47\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " Oct 30 18:37:54.483917 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.483094 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b41517e2-cc07-4530-8ec9-092c2a92ad47-istio-token\" (UniqueName: \"kubernetes.io/projected/b41517e2-cc07-4530-8ec9-092c2a92ad47-istio-token\") pod \"b41517e2-cc07-4530-8ec9-092c2a92ad47\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " Oct 30 18:37:54.483917 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.483138 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b41517e2-cc07-4530-8ec9-092c2a92ad47-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41517e2-cc07-4530-8ec9-092c2a92ad47-credential-socket\") pod \"b41517e2-cc07-4530-8ec9-092c2a92ad47\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " Oct 30 18:37:54.483917 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.483177 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b41517e2-cc07-4530-8ec9-092c2a92ad47-workload-socket" pod "b41517e2-cc07-4530-8ec9-092c2a92ad47" (UID: "b41517e2-cc07-4530-8ec9-092c2a92ad47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.483917 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.483200 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b41517e2-cc07-4530-8ec9-092c2a92ad47-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b41517e2-cc07-4530-8ec9-092c2a92ad47-istio-data\") pod \"b41517e2-cc07-4530-8ec9-092c2a92ad47\" (UID: \"b41517e2-cc07-4530-8ec9-092c2a92ad47\") " Oct 30 18:37:54.485226 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.484975 4455 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41517e2-cc07-4530-8ec9-092c2a92ad47-workload-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.485226 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.485008 4455 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b41517e2-cc07-4530-8ec9-092c2a92ad47-istiod-ca-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.485663 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.485478 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b41517e2-cc07-4530-8ec9-092c2a92ad47-istio-data" pod "b41517e2-cc07-4530-8ec9-092c2a92ad47" (UID: "b41517e2-cc07-4530-8ec9-092c2a92ad47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.485980 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.485796 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b41517e2-cc07-4530-8ec9-092c2a92ad47-credential-socket" pod "b41517e2-cc07-4530-8ec9-092c2a92ad47" (UID: "b41517e2-cc07-4530-8ec9-092c2a92ad47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.486450 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.486132 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b41517e2-cc07-4530-8ec9-092c2a92ad47-workload-certs" pod "b41517e2-cc07-4530-8ec9-092c2a92ad47" (UID: "b41517e2-cc07-4530-8ec9-092c2a92ad47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.516389 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.516338 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b41517e2-cc07-4530-8ec9-092c2a92ad47-istio-podinfo" pod "b41517e2-cc07-4530-8ec9-092c2a92ad47" (UID: "b41517e2-cc07-4530-8ec9-092c2a92ad47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:37:54.516513 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.516380 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b41517e2-cc07-4530-8ec9-092c2a92ad47-istio-token" pod "b41517e2-cc07-4530-8ec9-092c2a92ad47" (UID: "b41517e2-cc07-4530-8ec9-092c2a92ad47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:54.517487 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.517439 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b41517e2-cc07-4530-8ec9-092c2a92ad47-kube-api-access-nqfzv" pod "b41517e2-cc07-4530-8ec9-092c2a92ad47" (UID: "b41517e2-cc07-4530-8ec9-092c2a92ad47"). InnerVolumeSpecName "kube-api-access-nqfzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:54.520177 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.520002 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b41517e2-cc07-4530-8ec9-092c2a92ad47-istio-envoy" pod "b41517e2-cc07-4530-8ec9-092c2a92ad47" (UID: "b41517e2-cc07-4530-8ec9-092c2a92ad47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.587371 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.586879 4455 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b41517e2-cc07-4530-8ec9-092c2a92ad47-istio-envoy\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.587371 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.586919 4455 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b41517e2-cc07-4530-8ec9-092c2a92ad47-istio-podinfo\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.587371 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.586935 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqfzv\" (UniqueName: \"kubernetes.io/projected/b41517e2-cc07-4530-8ec9-092c2a92ad47-kube-api-access-nqfzv\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.587371 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.586950 4455 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b41517e2-cc07-4530-8ec9-092c2a92ad47-workload-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.587371 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.586964 4455 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b41517e2-cc07-4530-8ec9-092c2a92ad47-istio-token\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.587371 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.586976 4455 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41517e2-cc07-4530-8ec9-092c2a92ad47-credential-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.587371 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.586988 4455 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b41517e2-cc07-4530-8ec9-092c2a92ad47-istio-data\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.682407 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.682378 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" Oct 30 18:37:54.687773 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.687730 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8a6e230-4b8f-4d8d-af42-d6660522152e-istio-token\" (UniqueName: \"kubernetes.io/projected/a8a6e230-4b8f-4d8d-af42-d6660522152e-istio-token\") pod \"a8a6e230-4b8f-4d8d-af42-d6660522152e\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " Oct 30 18:37:54.687961 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.687803 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8a6e230-4b8f-4d8d-af42-d6660522152e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a6e230-4b8f-4d8d-af42-d6660522152e-credential-socket\") pod \"a8a6e230-4b8f-4d8d-af42-d6660522152e\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " Oct 30 18:37:54.687961 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.687924 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8a6e230-4b8f-4d8d-af42-d6660522152e-kube-api-access-htc5m\" (UniqueName: \"kubernetes.io/projected/a8a6e230-4b8f-4d8d-af42-d6660522152e-kube-api-access-htc5m\") pod \"a8a6e230-4b8f-4d8d-af42-d6660522152e\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " Oct 30 18:37:54.688097 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.688025 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8a6e230-4b8f-4d8d-af42-d6660522152e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8a6e230-4b8f-4d8d-af42-d6660522152e-istio-envoy\") pod \"a8a6e230-4b8f-4d8d-af42-d6660522152e\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " Oct 30 18:37:54.688097 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.688083 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a6e230-4b8f-4d8d-af42-d6660522152e-credential-socket" pod "a8a6e230-4b8f-4d8d-af42-d6660522152e" (UID: "a8a6e230-4b8f-4d8d-af42-d6660522152e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.688450 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.688417 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a6e230-4b8f-4d8d-af42-d6660522152e-workload-certs" pod "a8a6e230-4b8f-4d8d-af42-d6660522152e" (UID: "a8a6e230-4b8f-4d8d-af42-d6660522152e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.688637 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.688507 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8a6e230-4b8f-4d8d-af42-d6660522152e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8a6e230-4b8f-4d8d-af42-d6660522152e-workload-certs\") pod \"a8a6e230-4b8f-4d8d-af42-d6660522152e\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " Oct 30 18:37:54.688637 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.688579 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8a6e230-4b8f-4d8d-af42-d6660522152e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8a6e230-4b8f-4d8d-af42-d6660522152e-istio-podinfo\") pod \"a8a6e230-4b8f-4d8d-af42-d6660522152e\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " Oct 30 18:37:54.688850 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.688676 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8a6e230-4b8f-4d8d-af42-d6660522152e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8a6e230-4b8f-4d8d-af42-d6660522152e-istio-data\") pod \"a8a6e230-4b8f-4d8d-af42-d6660522152e\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " Oct 30 18:37:54.688850 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.688763 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8a6e230-4b8f-4d8d-af42-d6660522152e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a6e230-4b8f-4d8d-af42-d6660522152e-workload-socket\") pod \"a8a6e230-4b8f-4d8d-af42-d6660522152e\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " Oct 30 18:37:54.688964 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.688893 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8a6e230-4b8f-4d8d-af42-d6660522152e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8a6e230-4b8f-4d8d-af42-d6660522152e-istiod-ca-cert\") pod \"a8a6e230-4b8f-4d8d-af42-d6660522152e\" (UID: \"a8a6e230-4b8f-4d8d-af42-d6660522152e\") " Oct 30 18:37:54.690054 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.689455 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a6e230-4b8f-4d8d-af42-d6660522152e-workload-socket" pod "a8a6e230-4b8f-4d8d-af42-d6660522152e" (UID: "a8a6e230-4b8f-4d8d-af42-d6660522152e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.690054 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.689568 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a6e230-4b8f-4d8d-af42-d6660522152e-istio-data" pod "a8a6e230-4b8f-4d8d-af42-d6660522152e" (UID: "a8a6e230-4b8f-4d8d-af42-d6660522152e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.690054 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.689677 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8a6e230-4b8f-4d8d-af42-d6660522152e-istiod-ca-cert" pod "a8a6e230-4b8f-4d8d-af42-d6660522152e" (UID: "a8a6e230-4b8f-4d8d-af42-d6660522152e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:54.690054 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.689975 4455 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8a6e230-4b8f-4d8d-af42-d6660522152e-istiod-ca-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.690054 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.689998 4455 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a6e230-4b8f-4d8d-af42-d6660522152e-credential-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.690054 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.690014 4455 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8a6e230-4b8f-4d8d-af42-d6660522152e-workload-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.690054 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.690029 4455 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8a6e230-4b8f-4d8d-af42-d6660522152e-istio-data\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.690054 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.690044 4455 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a6e230-4b8f-4d8d-af42-d6660522152e-workload-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.690761 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.690609 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8a6e230-4b8f-4d8d-af42-d6660522152e-kube-api-access-htc5m" pod "a8a6e230-4b8f-4d8d-af42-d6660522152e" (UID: "a8a6e230-4b8f-4d8d-af42-d6660522152e"). InnerVolumeSpecName "kube-api-access-htc5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:54.691104 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.691073 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a6e230-4b8f-4d8d-af42-d6660522152e-istio-envoy" pod "a8a6e230-4b8f-4d8d-af42-d6660522152e" (UID: "a8a6e230-4b8f-4d8d-af42-d6660522152e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.691878 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.691858 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8a6e230-4b8f-4d8d-af42-d6660522152e-istio-podinfo" pod "a8a6e230-4b8f-4d8d-af42-d6660522152e" (UID: "a8a6e230-4b8f-4d8d-af42-d6660522152e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:37:54.692123 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.692109 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8a6e230-4b8f-4d8d-af42-d6660522152e-istio-token" pod "a8a6e230-4b8f-4d8d-af42-d6660522152e" (UID: "a8a6e230-4b8f-4d8d-af42-d6660522152e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:54.791878 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.791821 4455 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8a6e230-4b8f-4d8d-af42-d6660522152e-istio-token\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.791878 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.791869 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htc5m\" (UniqueName: \"kubernetes.io/projected/a8a6e230-4b8f-4d8d-af42-d6660522152e-kube-api-access-htc5m\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.791878 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.791880 4455 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8a6e230-4b8f-4d8d-af42-d6660522152e-istio-envoy\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.791878 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:54.791893 4455 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8a6e230-4b8f-4d8d-af42-d6660522152e-istio-podinfo\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:37:55.297449 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:55.297358 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" event={"ID":"b41517e2-cc07-4530-8ec9-092c2a92ad47","Type":"ContainerDied","Data":"1c6d6a053664ef57cae6df2c57976dc6f629cb80a4b269396431a670825a2e10"} Oct 30 18:37:55.297449 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:55.297427 4455 scope.go:122] "RemoveContainer" containerID="ce80d0cadfc75aac9e1f53d048259d4126674e32cbe065b2a660e32f08699c89" Oct 30 18:37:55.298082 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:55.297375 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv" Oct 30 18:37:55.299882 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:55.299700 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" Oct 30 18:37:55.299882 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:55.299697 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6" event={"ID":"a8a6e230-4b8f-4d8d-af42-d6660522152e","Type":"ContainerDied","Data":"416079477a8f875ad962365536d51229ae3c4bc8e49fb669cfe1d1f432753cf4"} Oct 30 18:37:55.312542 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:55.312521 4455 scope.go:122] "RemoveContainer" containerID="e2002cc6bfa68049e67590ef9ea1a59f8c518482ed44f52c2ff7d730d7e55671" Oct 30 18:37:55.415895 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:55.412660 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv"] Oct 30 18:37:55.418161 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:55.418132 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cxmbv"] Oct 30 18:37:55.439856 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:55.439806 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6"] Oct 30 18:37:55.444369 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:55.444316 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-s7wm6"] Oct 30 18:37:55.636863 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:55.636784 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8a6e230-4b8f-4d8d-af42-d6660522152e" path="/var/lib/kubelet/pods/a8a6e230-4b8f-4d8d-af42-d6660522152e/volumes" Oct 30 18:37:55.637371 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:37:55.637352 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b41517e2-cc07-4530-8ec9-092c2a92ad47" path="/var/lib/kubelet/pods/b41517e2-cc07-4530-8ec9-092c2a92ad47/volumes" Oct 30 18:38:00.423974 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:38:00.423873 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:38:00.533447 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:00.533408 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-6e8fca-7df8c6d488-xwlc2"] Oct 30 18:38:00.567384 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:00.566898 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-6e8fca-79548bc54c-5nw4l"] Oct 30 18:38:05.552390 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:05.543450 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-6e8fca-7df8c6d488-xwlc2" podUID="1b032444-b7c0-4d3c-8a07-aee2df84d8bd" containerName="payload-processing" containerID="cri-o://a1be86cf28df5221c774c69addf2792ded73fdc761656f0d8e96ad88f7acbb6f" gracePeriod=25 Oct 30 18:38:05.569382 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:05.568154 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6e8fca-79548bc54c-5nw4l" podUID="b32c8229-35fa-4eff-abf2-4d3b8713b1eb" containerName="payload-pre-processing" containerID="cri-o://c4ea9cec19b8c89b5f85f2cf502f07c1bbe6b4fa4e51fa1619385bec8f5069b0" gracePeriod=25 Oct 30 18:38:06.444352 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:06.442614 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:06.893291 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:06.893248 4455 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:38:06.895711 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:06.893899 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" podUID="7c9b7598-cec3-43ab-90cf-93e7b407d68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:07.447625 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:07.445221 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:07.447625 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:07.445891 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:38:08.286442 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:08.286404 4455 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:38:08.287050 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:08.287016 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" podUID="3daa8589-9def-486f-82ee-0c923a18b68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:08.288198 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:08.288171 4455 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:38:08.288405 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:08.288382 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" podUID="bbf4bc50-3670-40de-8159-3d8805fdd601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:08.453411 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:08.450177 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:08.676396 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:38:08.676287 4455 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.036s" Oct 30 18:38:09.762010 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:09.761962 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" containerID="cri-o://144822bba13c775f3ce8d2f977524f702498a6d4cf18a945bf8adb693cacafc8" gracePeriod=45 Oct 30 18:38:09.858349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:09.830846 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded" Oct 30 18:38:09.976361 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:09.861356 4455 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:38:09.976361 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:09.861596 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" podUID="d2c2d808-5274-484e-8520-1f88fa79fc47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:10.776706 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:10.776666 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": EOF" Oct 30 18:38:10.803593 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:10.803554 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": dial tcp 10.134.0.102:8000: connect: connection refused" Oct 30 18:38:11.253461 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:11.253421 4455 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:38:11.253854 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:11.253822 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" podUID="b933cb99-2e87-4707-a33b-de09b5c03536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:11.254424 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:11.253728 4455 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:38:11.254612 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:11.254589 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" podUID="1ee94ddb-0834-4ee6-835c-4363b672e1c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:11.320226 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:11.320188 4455 generic.go:356] "Generic (PLEG): container finished" podID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerID="144822bba13c775f3ce8d2f977524f702498a6d4cf18a945bf8adb693cacafc8" exitCode=0 Oct 30 18:38:11.320737 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:11.320710 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" event={"ID":"d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7","Type":"ContainerDied","Data":"144822bba13c775f3ce8d2f977524f702498a6d4cf18a945bf8adb693cacafc8"} Oct 30 18:38:11.428986 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:11.428943 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": dial tcp 10.134.0.102:8000: connect: connection refused" Oct 30 18:38:11.866555 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:11.866513 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6e8fca-79548bc54c-5nw4l" podUID="b32c8229-35fa-4eff-abf2-4d3b8713b1eb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:38:11.918540 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:11.918490 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-6e8fca-7df8c6d488-xwlc2" podUID="1b032444-b7c0-4d3c-8a07-aee2df84d8bd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:38:12.369361 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:12.367253 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" event={"ID":"d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7","Type":"ContainerDied","Data":"96b20b1cc872f6fad763dd32712cd93e850fad51ca70375a390f51d216d0660d"} Oct 30 18:38:12.369361 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:12.367306 4455 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96b20b1cc872f6fad763dd32712cd93e850fad51ca70375a390f51d216d0660d" Oct 30 18:38:12.385567 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:12.385531 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:38:12.441865 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:12.441783 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-tls-certs\") pod \"d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7\" (UID: \"d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7\") " Oct 30 18:38:12.441865 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:12.441829 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-model-cache\") pod \"d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7\" (UID: \"d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7\") " Oct 30 18:38:12.442106 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:12.441934 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-kube-api-access-xtstq\" (UniqueName: \"kubernetes.io/projected/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-kube-api-access-xtstq\") pod \"d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7\" (UID: \"d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7\") " Oct 30 18:38:12.442106 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:12.442029 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-tmp-dir\") pod \"d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7\" (UID: \"d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7\") " Oct 30 18:38:12.442215 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:12.442201 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-dshm\") pod \"d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7\" (UID: \"d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7\") " Oct 30 18:38:12.442294 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:12.442259 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-home\" (UniqueName: \"kubernetes.io/empty-dir/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-home\") pod \"d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7\" (UID: \"d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7\") " Oct 30 18:38:12.442444 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:12.442411 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-model-cache" pod "d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" (UID: "d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:12.442567 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:12.442542 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-tmp-dir" pod "d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" (UID: "d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:12.442778 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:12.442752 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-home" pod "d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" (UID: "d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:12.443039 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:12.443013 4455 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-home\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:12.443363 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:12.443043 4455 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-model-cache\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:12.443363 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:12.443060 4455 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-tmp-dir\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:12.446466 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:12.446436 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-dshm" pod "d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" (UID: "d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:12.446884 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:12.446861 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-kube-api-access-xtstq" pod "d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" (UID: "d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7"). InnerVolumeSpecName "kube-api-access-xtstq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:12.451729 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:12.451700 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-tls-certs" pod "d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" (UID: "d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:12.544799 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:12.544755 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtstq\" (UniqueName: \"kubernetes.io/projected/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-kube-api-access-xtstq\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:12.544799 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:12.544799 4455 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-dshm\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:12.544799 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:12.544814 4455 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7-tls-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:13.380484 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:13.380452 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb" Oct 30 18:38:13.446560 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:13.446527 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb"] Oct 30 18:38:13.453930 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:13.453900 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-6e8fca/distinct-2-w2-6e8fca-kserve-756b7687cf-xr6mb"] Oct 30 18:38:13.638999 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:13.638890 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" path="/var/lib/kubelet/pods/d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7/volumes" Oct 30 18:38:21.803435 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:21.803392 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6e8fca-79548bc54c-5nw4l" podUID="b32c8229-35fa-4eff-abf2-4d3b8713b1eb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:38:21.803879 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:21.803441 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-6e8fca-7df8c6d488-xwlc2" podUID="1b032444-b7c0-4d3c-8a07-aee2df84d8bd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:38:25.815019 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:25.814930 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-6e8fca-7df8c6d488-xwlc2" Oct 30 18:38:25.819090 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:25.819051 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b032444-b7c0-4d3c-8a07-aee2df84d8bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b032444-b7c0-4d3c-8a07-aee2df84d8bd-plugins-config-volume\") pod \"1b032444-b7c0-4d3c-8a07-aee2df84d8bd\" (UID: \"1b032444-b7c0-4d3c-8a07-aee2df84d8bd\") " Oct 30 18:38:25.819234 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:25.819158 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b032444-b7c0-4d3c-8a07-aee2df84d8bd-kube-api-access-w8gcs\" (UniqueName: \"kubernetes.io/projected/1b032444-b7c0-4d3c-8a07-aee2df84d8bd-kube-api-access-w8gcs\") pod \"1b032444-b7c0-4d3c-8a07-aee2df84d8bd\" (UID: \"1b032444-b7c0-4d3c-8a07-aee2df84d8bd\") " Oct 30 18:38:25.819571 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:25.819541 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b032444-b7c0-4d3c-8a07-aee2df84d8bd-plugins-config-volume" pod "1b032444-b7c0-4d3c-8a07-aee2df84d8bd" (UID: "1b032444-b7c0-4d3c-8a07-aee2df84d8bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:25.820009 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:25.819983 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b032444-b7c0-4d3c-8a07-aee2df84d8bd-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:25.827942 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:25.823563 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b032444-b7c0-4d3c-8a07-aee2df84d8bd-kube-api-access-w8gcs" pod "1b032444-b7c0-4d3c-8a07-aee2df84d8bd" (UID: "1b032444-b7c0-4d3c-8a07-aee2df84d8bd"). InnerVolumeSpecName "kube-api-access-w8gcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:25.898881 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:25.898840 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6e8fca-79548bc54c-5nw4l" Oct 30 18:38:25.920537 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:25.920502 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b32c8229-35fa-4eff-abf2-4d3b8713b1eb-kube-api-access-864nd\" (UniqueName: \"kubernetes.io/projected/b32c8229-35fa-4eff-abf2-4d3b8713b1eb-kube-api-access-864nd\") pod \"b32c8229-35fa-4eff-abf2-4d3b8713b1eb\" (UID: \"b32c8229-35fa-4eff-abf2-4d3b8713b1eb\") " Oct 30 18:38:25.920736 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:25.920587 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b32c8229-35fa-4eff-abf2-4d3b8713b1eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b32c8229-35fa-4eff-abf2-4d3b8713b1eb-plugins-config-volume\") pod \"b32c8229-35fa-4eff-abf2-4d3b8713b1eb\" (UID: \"b32c8229-35fa-4eff-abf2-4d3b8713b1eb\") " Oct 30 18:38:25.920973 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:25.920948 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b32c8229-35fa-4eff-abf2-4d3b8713b1eb-plugins-config-volume" pod "b32c8229-35fa-4eff-abf2-4d3b8713b1eb" (UID: "b32c8229-35fa-4eff-abf2-4d3b8713b1eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:25.921304 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:25.921284 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8gcs\" (UniqueName: \"kubernetes.io/projected/1b032444-b7c0-4d3c-8a07-aee2df84d8bd-kube-api-access-w8gcs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:25.921446 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:25.921312 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b32c8229-35fa-4eff-abf2-4d3b8713b1eb-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:25.923004 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:25.922980 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b32c8229-35fa-4eff-abf2-4d3b8713b1eb-kube-api-access-864nd" pod "b32c8229-35fa-4eff-abf2-4d3b8713b1eb" (UID: "b32c8229-35fa-4eff-abf2-4d3b8713b1eb"). InnerVolumeSpecName "kube-api-access-864nd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:26.089558 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:26.089424 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-864nd\" (UniqueName: \"kubernetes.io/projected/b32c8229-35fa-4eff-abf2-4d3b8713b1eb-kube-api-access-864nd\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:26.556390 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:26.556356 4455 generic.go:356] "Generic (PLEG): container finished" podID="b32c8229-35fa-4eff-abf2-4d3b8713b1eb" containerID="c4ea9cec19b8c89b5f85f2cf502f07c1bbe6b4fa4e51fa1619385bec8f5069b0" exitCode=0 Oct 30 18:38:26.556600 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:26.556403 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6e8fca-79548bc54c-5nw4l" Oct 30 18:38:26.556600 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:26.556437 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6e8fca-79548bc54c-5nw4l" event={"ID":"b32c8229-35fa-4eff-abf2-4d3b8713b1eb","Type":"ContainerDied","Data":"c4ea9cec19b8c89b5f85f2cf502f07c1bbe6b4fa4e51fa1619385bec8f5069b0"} Oct 30 18:38:26.556600 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:26.556512 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6e8fca-79548bc54c-5nw4l" event={"ID":"b32c8229-35fa-4eff-abf2-4d3b8713b1eb","Type":"ContainerDied","Data":"7cbd050f3850552529a1d3fa78a2e3fe088d846fc7eecf773d56a3e350f6c7ad"} Oct 30 18:38:26.556600 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:26.556543 4455 scope.go:122] "RemoveContainer" containerID="c4ea9cec19b8c89b5f85f2cf502f07c1bbe6b4fa4e51fa1619385bec8f5069b0" Oct 30 18:38:26.557994 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:26.557973 4455 generic.go:356] "Generic (PLEG): container finished" podID="1b032444-b7c0-4d3c-8a07-aee2df84d8bd" containerID="a1be86cf28df5221c774c69addf2792ded73fdc761656f0d8e96ad88f7acbb6f" exitCode=0 Oct 30 18:38:26.558111 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:26.558032 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-6e8fca-7df8c6d488-xwlc2" event={"ID":"1b032444-b7c0-4d3c-8a07-aee2df84d8bd","Type":"ContainerDied","Data":"a1be86cf28df5221c774c69addf2792ded73fdc761656f0d8e96ad88f7acbb6f"} Oct 30 18:38:26.558111 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:26.558038 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-6e8fca-7df8c6d488-xwlc2" Oct 30 18:38:26.558111 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:26.558066 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-6e8fca-7df8c6d488-xwlc2" event={"ID":"1b032444-b7c0-4d3c-8a07-aee2df84d8bd","Type":"ContainerDied","Data":"e7240a7fef6527a0868a3fdedd6bfb053caca928f7958fa321f7309e205d964f"} Oct 30 18:38:26.568822 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:26.568804 4455 scope.go:122] "RemoveContainer" containerID="c4ea9cec19b8c89b5f85f2cf502f07c1bbe6b4fa4e51fa1619385bec8f5069b0" Oct 30 18:38:26.569180 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:38:26.569134 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4ea9cec19b8c89b5f85f2cf502f07c1bbe6b4fa4e51fa1619385bec8f5069b0\": container with ID starting with c4ea9cec19b8c89b5f85f2cf502f07c1bbe6b4fa4e51fa1619385bec8f5069b0 not found: ID does not exist" containerID="c4ea9cec19b8c89b5f85f2cf502f07c1bbe6b4fa4e51fa1619385bec8f5069b0" Oct 30 18:38:26.569297 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:26.569188 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4ea9cec19b8c89b5f85f2cf502f07c1bbe6b4fa4e51fa1619385bec8f5069b0"} err="failed to get container status \"c4ea9cec19b8c89b5f85f2cf502f07c1bbe6b4fa4e51fa1619385bec8f5069b0\": rpc error: code = NotFound desc = could not find container \"c4ea9cec19b8c89b5f85f2cf502f07c1bbe6b4fa4e51fa1619385bec8f5069b0\": container with ID starting with c4ea9cec19b8c89b5f85f2cf502f07c1bbe6b4fa4e51fa1619385bec8f5069b0 not found: ID does not exist" Oct 30 18:38:26.569297 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:26.569214 4455 scope.go:122] "RemoveContainer" containerID="a1be86cf28df5221c774c69addf2792ded73fdc761656f0d8e96ad88f7acbb6f" Oct 30 18:38:26.591138 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:26.591118 4455 scope.go:122] "RemoveContainer" containerID="a1be86cf28df5221c774c69addf2792ded73fdc761656f0d8e96ad88f7acbb6f" Oct 30 18:38:26.591525 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:38:26.591504 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1be86cf28df5221c774c69addf2792ded73fdc761656f0d8e96ad88f7acbb6f\": container with ID starting with a1be86cf28df5221c774c69addf2792ded73fdc761656f0d8e96ad88f7acbb6f not found: ID does not exist" containerID="a1be86cf28df5221c774c69addf2792ded73fdc761656f0d8e96ad88f7acbb6f" Oct 30 18:38:26.591625 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:26.591534 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1be86cf28df5221c774c69addf2792ded73fdc761656f0d8e96ad88f7acbb6f"} err="failed to get container status \"a1be86cf28df5221c774c69addf2792ded73fdc761656f0d8e96ad88f7acbb6f\": rpc error: code = NotFound desc = could not find container \"a1be86cf28df5221c774c69addf2792ded73fdc761656f0d8e96ad88f7acbb6f\": container with ID starting with a1be86cf28df5221c774c69addf2792ded73fdc761656f0d8e96ad88f7acbb6f not found: ID does not exist" Oct 30 18:38:26.656351 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:26.656304 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-6e8fca-79548bc54c-5nw4l"] Oct 30 18:38:26.660381 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:26.660358 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-6e8fca-79548bc54c-5nw4l"] Oct 30 18:38:26.717234 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:26.717201 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-6e8fca-7df8c6d488-xwlc2"] Oct 30 18:38:26.721734 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:26.721713 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-6e8fca-7df8c6d488-xwlc2"] Oct 30 18:38:27.639717 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:27.639682 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b032444-b7c0-4d3c-8a07-aee2df84d8bd" path="/var/lib/kubelet/pods/1b032444-b7c0-4d3c-8a07-aee2df84d8bd/volumes" Oct 30 18:38:27.640252 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:27.640231 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b32c8229-35fa-4eff-abf2-4d3b8713b1eb" path="/var/lib/kubelet/pods/b32c8229-35fa-4eff-abf2-4d3b8713b1eb/volumes" Oct 30 18:38:30.386654 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:38:30.386610 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:38:31.295963 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:31.295932 4455 scope.go:122] "RemoveContainer" containerID="144822bba13c775f3ce8d2f977524f702498a6d4cf18a945bf8adb693cacafc8" Oct 30 18:38:40.720115 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:40.720067 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr"] Oct 30 18:38:40.720970 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:40.720371 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr" podUID="038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5" containerName="maas-api" containerID="cri-o://7697d7815fff393e63570d60961c041ed77dbf6235f37519ef6b8e30b35da0bd" gracePeriod=30 Oct 30 18:38:41.369862 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:41.361569 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr" Oct 30 18:38:41.534272 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:41.533735 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5-kube-api-access-49qbf\" (UniqueName: \"kubernetes.io/projected/038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5-kube-api-access-49qbf\") pod \"038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5\" (UID: \"038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5\") " Oct 30 18:38:41.534272 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:41.533952 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5-openshift-service-ca\") pod \"038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5\" (UID: \"038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5\") " Oct 30 18:38:41.534272 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:41.534003 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5-metrics-certs\") pod \"038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5\" (UID: \"038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5\") " Oct 30 18:38:41.534272 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:41.534039 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5-maas-api-tls\") pod \"038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5\" (UID: \"038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5\") " Oct 30 18:38:41.534910 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:41.534388 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5-openshift-service-ca" pod "038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5" (UID: "038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:41.535120 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:41.535095 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:41.536576 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:41.536541 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5-metrics-certs" pod "038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5" (UID: "038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:41.537064 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:41.537024 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5-maas-api-tls" pod "038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5" (UID: "038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:41.537391 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:41.537370 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5-kube-api-access-49qbf" pod "038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5" (UID: "038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5"). InnerVolumeSpecName "kube-api-access-49qbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:41.665829 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:41.665072 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:41.665829 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:41.665103 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:41.665829 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:41.665117 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49qbf\" (UniqueName: \"kubernetes.io/projected/038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5-kube-api-access-49qbf\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:41.778809 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:41.778766 4455 generic.go:356] "Generic (PLEG): container finished" podID="038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5" containerID="7697d7815fff393e63570d60961c041ed77dbf6235f37519ef6b8e30b35da0bd" exitCode=0 Oct 30 18:38:41.779399 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:41.779365 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr" event={"ID":"038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5","Type":"ContainerDied","Data":"7697d7815fff393e63570d60961c041ed77dbf6235f37519ef6b8e30b35da0bd"} Oct 30 18:38:41.779604 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:41.779581 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr" event={"ID":"038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5","Type":"ContainerDied","Data":"3615b9710c958f63f5795eb60cee6e31476b9d78f155d01640739f07e2a14cd5"} Oct 30 18:38:41.779751 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:41.779736 4455 scope.go:122] "RemoveContainer" containerID="7697d7815fff393e63570d60961c041ed77dbf6235f37519ef6b8e30b35da0bd" Oct 30 18:38:41.786618 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:41.786418 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr" Oct 30 18:38:41.864001 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:41.863980 4455 scope.go:122] "RemoveContainer" containerID="7697d7815fff393e63570d60961c041ed77dbf6235f37519ef6b8e30b35da0bd" Oct 30 18:38:41.864521 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:38:41.864499 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7697d7815fff393e63570d60961c041ed77dbf6235f37519ef6b8e30b35da0bd\": container with ID starting with 7697d7815fff393e63570d60961c041ed77dbf6235f37519ef6b8e30b35da0bd not found: ID does not exist" containerID="7697d7815fff393e63570d60961c041ed77dbf6235f37519ef6b8e30b35da0bd" Oct 30 18:38:41.864654 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:41.864636 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7697d7815fff393e63570d60961c041ed77dbf6235f37519ef6b8e30b35da0bd"} err="failed to get container status \"7697d7815fff393e63570d60961c041ed77dbf6235f37519ef6b8e30b35da0bd\": rpc error: code = NotFound desc = could not find container \"7697d7815fff393e63570d60961c041ed77dbf6235f37519ef6b8e30b35da0bd\": container with ID starting with 7697d7815fff393e63570d60961c041ed77dbf6235f37519ef6b8e30b35da0bd not found: ID does not exist" Oct 30 18:38:41.894980 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:41.894951 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr"] Oct 30 18:38:41.917087 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:41.916999 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-6e8fca-85b94ddbc-82jcr"] Oct 30 18:38:43.105336 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:43.105288 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-58d08b-c58b968db-9f7zq"] Oct 30 18:38:43.190220 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:43.190185 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-58d08b-866bd6bfff-8vjgc"] Oct 30 18:38:43.636432 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:43.636397 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5" path="/var/lib/kubelet/pods/038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5/volumes" Oct 30 18:38:44.282760 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:44.282717 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a1a31f-5858ff7c48-xz2tp"] Oct 30 18:38:44.365956 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:44.365879 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-a1a31f-6dcdff67b7-gl4jg"] Oct 30 18:38:48.106377 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:48.106311 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-58d08b-c58b968db-9f7zq" podUID="03cfd14d-26f3-4294-a58d-9e47ad8b1b67" containerName="payload-processing" containerID="cri-o://05433e05a88bd86a48bc23071a6045719e83df925ba2838a843b6225b977ee40" gracePeriod=25 Oct 30 18:38:48.191576 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:48.191519 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-58d08b-866bd6bfff-8vjgc" podUID="baac9ab5-2caa-4596-a802-46062f6fcb4c" containerName="payload-pre-processing" containerID="cri-o://712e1f01575964bb17183355ffc5ab13f402fa4d7a934c34ff2d92b9fc36e607" gracePeriod=25 Oct 30 18:38:48.328971 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:38:48.328939 4455 logging.go:55] [core] [Channel #9365 SubChannel #9366]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.213:50052", ServerName: "10.134.0.213:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.213:50052: connect: connection refused" Oct 30 18:38:48.329175 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:38:48.329052 4455 logging.go:55] [core] [Channel #9364 SubChannel #9367]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.212:50052", ServerName: "10.134.0.212:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.212:50052: connect: connection refused" Oct 30 18:38:48.546910 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:48.546879 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-58d08b-c58b968db-9f7zq" Oct 30 18:38:48.590718 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:48.590684 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-58d08b-866bd6bfff-8vjgc" Oct 30 18:38:48.618712 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:48.618678 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03cfd14d-26f3-4294-a58d-9e47ad8b1b67-kube-api-access-jllz4\" (UniqueName: \"kubernetes.io/projected/03cfd14d-26f3-4294-a58d-9e47ad8b1b67-kube-api-access-jllz4\") pod \"03cfd14d-26f3-4294-a58d-9e47ad8b1b67\" (UID: \"03cfd14d-26f3-4294-a58d-9e47ad8b1b67\") " Oct 30 18:38:48.618897 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:48.618748 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03cfd14d-26f3-4294-a58d-9e47ad8b1b67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03cfd14d-26f3-4294-a58d-9e47ad8b1b67-plugins-config-volume\") pod \"03cfd14d-26f3-4294-a58d-9e47ad8b1b67\" (UID: \"03cfd14d-26f3-4294-a58d-9e47ad8b1b67\") " Oct 30 18:38:48.620125 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:48.619734 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03cfd14d-26f3-4294-a58d-9e47ad8b1b67-plugins-config-volume" pod "03cfd14d-26f3-4294-a58d-9e47ad8b1b67" (UID: "03cfd14d-26f3-4294-a58d-9e47ad8b1b67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:48.620125 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:48.619962 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03cfd14d-26f3-4294-a58d-9e47ad8b1b67-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:48.621432 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:48.621404 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03cfd14d-26f3-4294-a58d-9e47ad8b1b67-kube-api-access-jllz4" pod "03cfd14d-26f3-4294-a58d-9e47ad8b1b67" (UID: "03cfd14d-26f3-4294-a58d-9e47ad8b1b67"). InnerVolumeSpecName "kube-api-access-jllz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:48.720902 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:48.720871 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baac9ab5-2caa-4596-a802-46062f6fcb4c-kube-api-access-pv5zg\" (UniqueName: \"kubernetes.io/projected/baac9ab5-2caa-4596-a802-46062f6fcb4c-kube-api-access-pv5zg\") pod \"baac9ab5-2caa-4596-a802-46062f6fcb4c\" (UID: \"baac9ab5-2caa-4596-a802-46062f6fcb4c\") " Oct 30 18:38:48.721133 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:48.721114 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baac9ab5-2caa-4596-a802-46062f6fcb4c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baac9ab5-2caa-4596-a802-46062f6fcb4c-plugins-config-volume\") pod \"baac9ab5-2caa-4596-a802-46062f6fcb4c\" (UID: \"baac9ab5-2caa-4596-a802-46062f6fcb4c\") " Oct 30 18:38:48.721789 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:48.721763 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baac9ab5-2caa-4596-a802-46062f6fcb4c-plugins-config-volume" pod "baac9ab5-2caa-4596-a802-46062f6fcb4c" (UID: "baac9ab5-2caa-4596-a802-46062f6fcb4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:48.722011 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:48.721992 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jllz4\" (UniqueName: \"kubernetes.io/projected/03cfd14d-26f3-4294-a58d-9e47ad8b1b67-kube-api-access-jllz4\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:48.723551 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:48.723527 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baac9ab5-2caa-4596-a802-46062f6fcb4c-kube-api-access-pv5zg" pod "baac9ab5-2caa-4596-a802-46062f6fcb4c" (UID: "baac9ab5-2caa-4596-a802-46062f6fcb4c"). InnerVolumeSpecName "kube-api-access-pv5zg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:48.823764 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:48.823728 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baac9ab5-2caa-4596-a802-46062f6fcb4c-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:48.823764 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:48.823758 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv5zg\" (UniqueName: \"kubernetes.io/projected/baac9ab5-2caa-4596-a802-46062f6fcb4c-kube-api-access-pv5zg\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:48.838105 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:48.838075 4455 generic.go:356] "Generic (PLEG): container finished" podID="03cfd14d-26f3-4294-a58d-9e47ad8b1b67" containerID="05433e05a88bd86a48bc23071a6045719e83df925ba2838a843b6225b977ee40" exitCode=0 Oct 30 18:38:48.838254 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:48.838131 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-58d08b-c58b968db-9f7zq" Oct 30 18:38:48.838254 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:48.838160 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-58d08b-c58b968db-9f7zq" event={"ID":"03cfd14d-26f3-4294-a58d-9e47ad8b1b67","Type":"ContainerDied","Data":"05433e05a88bd86a48bc23071a6045719e83df925ba2838a843b6225b977ee40"} Oct 30 18:38:48.838254 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:48.838216 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-58d08b-c58b968db-9f7zq" event={"ID":"03cfd14d-26f3-4294-a58d-9e47ad8b1b67","Type":"ContainerDied","Data":"6c8865329862b160bc2d0f5ec37300ca06e5d31975a62bf741a12644737388a8"} Oct 30 18:38:48.838254 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:48.838228 4455 scope.go:122] "RemoveContainer" containerID="05433e05a88bd86a48bc23071a6045719e83df925ba2838a843b6225b977ee40" Oct 30 18:38:48.840409 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:48.840385 4455 generic.go:356] "Generic (PLEG): container finished" podID="baac9ab5-2caa-4596-a802-46062f6fcb4c" containerID="712e1f01575964bb17183355ffc5ab13f402fa4d7a934c34ff2d92b9fc36e607" exitCode=0 Oct 30 18:38:48.840632 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:48.840609 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-58d08b-866bd6bfff-8vjgc" Oct 30 18:38:48.841290 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:48.841260 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-58d08b-866bd6bfff-8vjgc" event={"ID":"baac9ab5-2caa-4596-a802-46062f6fcb4c","Type":"ContainerDied","Data":"712e1f01575964bb17183355ffc5ab13f402fa4d7a934c34ff2d92b9fc36e607"} Oct 30 18:38:48.841965 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:48.841403 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-58d08b-866bd6bfff-8vjgc" event={"ID":"baac9ab5-2caa-4596-a802-46062f6fcb4c","Type":"ContainerDied","Data":"dc6587da2bd2eeecaa56d571929dacafed14ad5f11a9f5d8c2fcf18dca0fee1f"} Oct 30 18:38:48.856000 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:48.855982 4455 scope.go:122] "RemoveContainer" containerID="05433e05a88bd86a48bc23071a6045719e83df925ba2838a843b6225b977ee40" Oct 30 18:38:48.856504 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:38:48.856474 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05433e05a88bd86a48bc23071a6045719e83df925ba2838a843b6225b977ee40\": container with ID starting with 05433e05a88bd86a48bc23071a6045719e83df925ba2838a843b6225b977ee40 not found: ID does not exist" containerID="05433e05a88bd86a48bc23071a6045719e83df925ba2838a843b6225b977ee40" Oct 30 18:38:48.856612 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:48.856516 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05433e05a88bd86a48bc23071a6045719e83df925ba2838a843b6225b977ee40"} err="failed to get container status \"05433e05a88bd86a48bc23071a6045719e83df925ba2838a843b6225b977ee40\": rpc error: code = NotFound desc = could not find container \"05433e05a88bd86a48bc23071a6045719e83df925ba2838a843b6225b977ee40\": container with ID starting with 05433e05a88bd86a48bc23071a6045719e83df925ba2838a843b6225b977ee40 not found: ID does not exist" Oct 30 18:38:48.856612 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:48.856541 4455 scope.go:122] "RemoveContainer" containerID="712e1f01575964bb17183355ffc5ab13f402fa4d7a934c34ff2d92b9fc36e607" Oct 30 18:38:48.894952 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:48.894919 4455 scope.go:122] "RemoveContainer" containerID="712e1f01575964bb17183355ffc5ab13f402fa4d7a934c34ff2d92b9fc36e607" Oct 30 18:38:48.895368 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:38:48.895304 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"712e1f01575964bb17183355ffc5ab13f402fa4d7a934c34ff2d92b9fc36e607\": container with ID starting with 712e1f01575964bb17183355ffc5ab13f402fa4d7a934c34ff2d92b9fc36e607 not found: ID does not exist" containerID="712e1f01575964bb17183355ffc5ab13f402fa4d7a934c34ff2d92b9fc36e607" Oct 30 18:38:48.895439 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:48.895379 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"712e1f01575964bb17183355ffc5ab13f402fa4d7a934c34ff2d92b9fc36e607"} err="failed to get container status \"712e1f01575964bb17183355ffc5ab13f402fa4d7a934c34ff2d92b9fc36e607\": rpc error: code = NotFound desc = could not find container \"712e1f01575964bb17183355ffc5ab13f402fa4d7a934c34ff2d92b9fc36e607\": container with ID starting with 712e1f01575964bb17183355ffc5ab13f402fa4d7a934c34ff2d92b9fc36e607 not found: ID does not exist" Oct 30 18:38:48.935370 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:48.935299 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-58d08b-866bd6bfff-8vjgc"] Oct 30 18:38:48.939385 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:48.939360 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-58d08b-866bd6bfff-8vjgc"] Oct 30 18:38:48.962301 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:48.962272 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x"] Oct 30 18:38:48.962523 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:48.962502 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" podUID="e972725e-489d-4f2a-a734-27d7eb570858" containerName="istio-proxy" containerID="cri-o://946f14782c693823bc55858538440fc8959065207e645894ca6d65a811047082" gracePeriod=30 Oct 30 18:38:49.006664 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.006582 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp"] Oct 30 18:38:49.006889 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.006853 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" podUID="ac80ccea-a611-4de1-951c-479ad3d827a7" containerName="istio-proxy" containerID="cri-o://36e7d4a4bb7018d8514f41830ca5589fbd87445555b98301f09584b942a62f0f" gracePeriod=30 Oct 30 18:38:49.037349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.015985 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26"] Oct 30 18:38:49.037349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.016270 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" podUID="93b710b6-b5a8-4fe0-ac70-861dd036b70d" containerName="istio-proxy" containerID="cri-o://b693433aac4f98920aa85577a8ea16b04a62c0737ed7789f243a93630cc5955d" gracePeriod=30 Oct 30 18:38:49.168345 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.167943 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-58d08b-c58b968db-9f7zq"] Oct 30 18:38:49.185364 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.169593 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-58d08b-c58b968db-9f7zq"] Oct 30 18:38:49.283786 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.283684 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-a1a31f-5858ff7c48-xz2tp" podUID="502021d0-cc42-404e-a0f7-67f7392c82bf" containerName="payload-processing" containerID="cri-o://f75b149c6027c34cb527b3d8ada9717dc241c6b7aa391361cd2c4cb1770a090a" gracePeriod=25 Oct 30 18:38:49.328369 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.327573 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-autores-58d08b-866bd6bfff-8vjgc" podUID="baac9ab5-2caa-4596-a802-46062f6fcb4c" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.212:50052\" within 1s: context deadline exceeded" Oct 30 18:38:49.328369 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.327573 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-autores-58d08b-c58b968db-9f7zq" podUID="03cfd14d-26f3-4294-a58d-9e47ad8b1b67" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.213:50052\" within 1s: context deadline exceeded" Oct 30 18:38:49.367202 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.367162 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a1a31f-6dcdff67b7-gl4jg" podUID="1ae8cd58-07cf-40ae-9421-d2c91aee0379" containerName="payload-pre-processing" containerID="cri-o://8b51075b6d9ea930c3b2349c2d9a8aadef5a8a9c1674ac7a822914ef62502b8f" gracePeriod=25 Oct 30 18:38:49.671396 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.671357 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03cfd14d-26f3-4294-a58d-9e47ad8b1b67" path="/var/lib/kubelet/pods/03cfd14d-26f3-4294-a58d-9e47ad8b1b67/volumes" Oct 30 18:38:49.671895 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.671879 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baac9ab5-2caa-4596-a802-46062f6fcb4c" path="/var/lib/kubelet/pods/baac9ab5-2caa-4596-a802-46062f6fcb4c/volumes" Oct 30 18:38:49.744017 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.743976 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-a1a31f-5858ff7c48-xz2tp" Oct 30 18:38:49.786407 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.786382 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a1a31f-6dcdff67b7-gl4jg" Oct 30 18:38:49.854406 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.854307 4455 generic.go:356] "Generic (PLEG): container finished" podID="502021d0-cc42-404e-a0f7-67f7392c82bf" containerID="f75b149c6027c34cb527b3d8ada9717dc241c6b7aa391361cd2c4cb1770a090a" exitCode=0 Oct 30 18:38:49.854684 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.854655 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-a1a31f-5858ff7c48-xz2tp" event={"ID":"502021d0-cc42-404e-a0f7-67f7392c82bf","Type":"ContainerDied","Data":"f75b149c6027c34cb527b3d8ada9717dc241c6b7aa391361cd2c4cb1770a090a"} Oct 30 18:38:49.854824 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.854722 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-a1a31f-5858ff7c48-xz2tp" event={"ID":"502021d0-cc42-404e-a0f7-67f7392c82bf","Type":"ContainerDied","Data":"046527b8a3238903bfbf7916aed75e6fdbe9d603115940e3db50cbb82f6b5685"} Oct 30 18:38:49.854824 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.854746 4455 scope.go:122] "RemoveContainer" containerID="f75b149c6027c34cb527b3d8ada9717dc241c6b7aa391361cd2c4cb1770a090a" Oct 30 18:38:49.854824 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.854824 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-a1a31f-5858ff7c48-xz2tp" Oct 30 18:38:49.883814 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.883772 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/502021d0-cc42-404e-a0f7-67f7392c82bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/502021d0-cc42-404e-a0f7-67f7392c82bf-plugins-config-volume\") pod \"502021d0-cc42-404e-a0f7-67f7392c82bf\" (UID: \"502021d0-cc42-404e-a0f7-67f7392c82bf\") " Oct 30 18:38:49.883966 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.883952 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ae8cd58-07cf-40ae-9421-d2c91aee0379-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae8cd58-07cf-40ae-9421-d2c91aee0379-plugins-config-volume\") pod \"1ae8cd58-07cf-40ae-9421-d2c91aee0379\" (UID: \"1ae8cd58-07cf-40ae-9421-d2c91aee0379\") " Oct 30 18:38:49.884032 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.883981 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/502021d0-cc42-404e-a0f7-67f7392c82bf-kube-api-access-k62vt\" (UniqueName: \"kubernetes.io/projected/502021d0-cc42-404e-a0f7-67f7392c82bf-kube-api-access-k62vt\") pod \"502021d0-cc42-404e-a0f7-67f7392c82bf\" (UID: \"502021d0-cc42-404e-a0f7-67f7392c82bf\") " Oct 30 18:38:49.884483 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.884157 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ae8cd58-07cf-40ae-9421-d2c91aee0379-kube-api-access-4dcx6\" (UniqueName: \"kubernetes.io/projected/1ae8cd58-07cf-40ae-9421-d2c91aee0379-kube-api-access-4dcx6\") pod \"1ae8cd58-07cf-40ae-9421-d2c91aee0379\" (UID: \"1ae8cd58-07cf-40ae-9421-d2c91aee0379\") " Oct 30 18:38:49.884672 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.884646 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ae8cd58-07cf-40ae-9421-d2c91aee0379-plugins-config-volume" pod "1ae8cd58-07cf-40ae-9421-d2c91aee0379" (UID: "1ae8cd58-07cf-40ae-9421-d2c91aee0379"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:49.884917 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.884899 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae8cd58-07cf-40ae-9421-d2c91aee0379-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:49.885267 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.885247 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/502021d0-cc42-404e-a0f7-67f7392c82bf-plugins-config-volume" pod "502021d0-cc42-404e-a0f7-67f7392c82bf" (UID: "502021d0-cc42-404e-a0f7-67f7392c82bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:49.887826 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.887801 4455 generic.go:356] "Generic (PLEG): container finished" podID="1ae8cd58-07cf-40ae-9421-d2c91aee0379" containerID="8b51075b6d9ea930c3b2349c2d9a8aadef5a8a9c1674ac7a822914ef62502b8f" exitCode=0 Oct 30 18:38:49.887939 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.887918 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a1a31f-6dcdff67b7-gl4jg" event={"ID":"1ae8cd58-07cf-40ae-9421-d2c91aee0379","Type":"ContainerDied","Data":"8b51075b6d9ea930c3b2349c2d9a8aadef5a8a9c1674ac7a822914ef62502b8f"} Oct 30 18:38:49.888006 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.887980 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a1a31f-6dcdff67b7-gl4jg" event={"ID":"1ae8cd58-07cf-40ae-9421-d2c91aee0379","Type":"ContainerDied","Data":"c39dcdcb73103cf812ab6883ee67f80f3ab88004f053385cf741eb41d7d94f50"} Oct 30 18:38:49.888078 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.888068 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a1a31f-6dcdff67b7-gl4jg" Oct 30 18:38:49.888171 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.888144 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae8cd58-07cf-40ae-9421-d2c91aee0379-kube-api-access-4dcx6" pod "1ae8cd58-07cf-40ae-9421-d2c91aee0379" (UID: "1ae8cd58-07cf-40ae-9421-d2c91aee0379"). InnerVolumeSpecName "kube-api-access-4dcx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:49.889420 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.889391 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/502021d0-cc42-404e-a0f7-67f7392c82bf-kube-api-access-k62vt" pod "502021d0-cc42-404e-a0f7-67f7392c82bf" (UID: "502021d0-cc42-404e-a0f7-67f7392c82bf"). InnerVolumeSpecName "kube-api-access-k62vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:49.896262 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.896243 4455 scope.go:122] "RemoveContainer" containerID="f75b149c6027c34cb527b3d8ada9717dc241c6b7aa391361cd2c4cb1770a090a" Oct 30 18:38:49.896740 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:38:49.896699 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f75b149c6027c34cb527b3d8ada9717dc241c6b7aa391361cd2c4cb1770a090a\": container with ID starting with f75b149c6027c34cb527b3d8ada9717dc241c6b7aa391361cd2c4cb1770a090a not found: ID does not exist" containerID="f75b149c6027c34cb527b3d8ada9717dc241c6b7aa391361cd2c4cb1770a090a" Oct 30 18:38:49.896831 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.896750 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f75b149c6027c34cb527b3d8ada9717dc241c6b7aa391361cd2c4cb1770a090a"} err="failed to get container status \"f75b149c6027c34cb527b3d8ada9717dc241c6b7aa391361cd2c4cb1770a090a\": rpc error: code = NotFound desc = could not find container \"f75b149c6027c34cb527b3d8ada9717dc241c6b7aa391361cd2c4cb1770a090a\": container with ID starting with f75b149c6027c34cb527b3d8ada9717dc241c6b7aa391361cd2c4cb1770a090a not found: ID does not exist" Oct 30 18:38:49.896831 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.896775 4455 scope.go:122] "RemoveContainer" containerID="8b51075b6d9ea930c3b2349c2d9a8aadef5a8a9c1674ac7a822914ef62502b8f" Oct 30 18:38:49.980904 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.977157 4455 scope.go:122] "RemoveContainer" containerID="8b51075b6d9ea930c3b2349c2d9a8aadef5a8a9c1674ac7a822914ef62502b8f" Oct 30 18:38:49.984291 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:38:49.984251 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b51075b6d9ea930c3b2349c2d9a8aadef5a8a9c1674ac7a822914ef62502b8f\": container with ID starting with 8b51075b6d9ea930c3b2349c2d9a8aadef5a8a9c1674ac7a822914ef62502b8f not found: ID does not exist" containerID="8b51075b6d9ea930c3b2349c2d9a8aadef5a8a9c1674ac7a822914ef62502b8f" Oct 30 18:38:49.984556 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.984526 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b51075b6d9ea930c3b2349c2d9a8aadef5a8a9c1674ac7a822914ef62502b8f"} err="failed to get container status \"8b51075b6d9ea930c3b2349c2d9a8aadef5a8a9c1674ac7a822914ef62502b8f\": rpc error: code = NotFound desc = could not find container \"8b51075b6d9ea930c3b2349c2d9a8aadef5a8a9c1674ac7a822914ef62502b8f\": container with ID starting with 8b51075b6d9ea930c3b2349c2d9a8aadef5a8a9c1674ac7a822914ef62502b8f not found: ID does not exist" Oct 30 18:38:49.986803 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.986779 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dcx6\" (UniqueName: \"kubernetes.io/projected/1ae8cd58-07cf-40ae-9421-d2c91aee0379-kube-api-access-4dcx6\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:49.986973 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.986960 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/502021d0-cc42-404e-a0f7-67f7392c82bf-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:49.987058 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:49.987048 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k62vt\" (UniqueName: \"kubernetes.io/projected/502021d0-cc42-404e-a0f7-67f7392c82bf-kube-api-access-k62vt\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:50.215689 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:50.215664 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a1a31f-5858ff7c48-xz2tp"] Oct 30 18:38:50.228268 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:50.228239 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a1a31f-5858ff7c48-xz2tp"] Oct 30 18:38:50.276126 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:50.276096 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-a1a31f-6dcdff67b7-gl4jg"] Oct 30 18:38:50.281683 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:50.281655 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-a1a31f-6dcdff67b7-gl4jg"] Oct 30 18:38:51.703822 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:51.701339 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ae8cd58-07cf-40ae-9421-d2c91aee0379" path="/var/lib/kubelet/pods/1ae8cd58-07cf-40ae-9421-d2c91aee0379/volumes" Oct 30 18:38:51.703822 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:51.702347 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="502021d0-cc42-404e-a0f7-67f7392c82bf" path="/var/lib/kubelet/pods/502021d0-cc42-404e-a0f7-67f7392c82bf/volumes" Oct 30 18:38:52.026648 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:52.026612 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j"] Oct 30 18:38:52.027167 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:52.027137 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" podUID="7c9b7598-cec3-43ab-90cf-93e7b407d68e" containerName="istio-proxy" containerID="cri-o://524f212977033a1bd2de88b5b493663bfe5755c1ed253ef4a3df7b5be1378a5b" gracePeriod=30 Oct 30 18:38:52.056734 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:52.056706 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98"] Oct 30 18:38:52.056907 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:52.056890 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" podUID="bbf4bc50-3670-40de-8159-3d8805fdd601" containerName="istio-proxy" containerID="cri-o://f7d22f95cf4835ebe39cfc8659ce9bf4786d0f2fa16aafda6f92cbf7fcc80cac" gracePeriod=30 Oct 30 18:38:52.067346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:52.063502 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf"] Oct 30 18:38:52.067346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:52.063722 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" podUID="3daa8589-9def-486f-82ee-0c923a18b68b" containerName="istio-proxy" containerID="cri-o://9736a79721eca5b603182cff32bfc216e3c642b536bb8178a0f7d54b8017d80f" gracePeriod=30 Oct 30 18:38:52.303480 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:52.296159 4455 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:38:52.303480 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:52.296227 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" podUID="3daa8589-9def-486f-82ee-0c923a18b68b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:38:54.313010 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.312977 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" Oct 30 18:38:54.407741 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.407700 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e972725e-489d-4f2a-a734-27d7eb570858-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e972725e-489d-4f2a-a734-27d7eb570858-credential-socket\") pod \"e972725e-489d-4f2a-a734-27d7eb570858\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " Oct 30 18:38:54.407911 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.407776 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e972725e-489d-4f2a-a734-27d7eb570858-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e972725e-489d-4f2a-a734-27d7eb570858-istiod-ca-cert\") pod \"e972725e-489d-4f2a-a734-27d7eb570858\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " Oct 30 18:38:54.407911 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.407899 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e972725e-489d-4f2a-a734-27d7eb570858-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e972725e-489d-4f2a-a734-27d7eb570858-istio-data\") pod \"e972725e-489d-4f2a-a734-27d7eb570858\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " Oct 30 18:38:54.408030 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.407978 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e972725e-489d-4f2a-a734-27d7eb570858-kube-api-access-r6sqd\" (UniqueName: \"kubernetes.io/projected/e972725e-489d-4f2a-a734-27d7eb570858-kube-api-access-r6sqd\") pod \"e972725e-489d-4f2a-a734-27d7eb570858\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " Oct 30 18:38:54.408094 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.408032 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e972725e-489d-4f2a-a734-27d7eb570858-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e972725e-489d-4f2a-a734-27d7eb570858-istio-podinfo\") pod \"e972725e-489d-4f2a-a734-27d7eb570858\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " Oct 30 18:38:54.408232 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.408209 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e972725e-489d-4f2a-a734-27d7eb570858-istio-token\" (UniqueName: \"kubernetes.io/projected/e972725e-489d-4f2a-a734-27d7eb570858-istio-token\") pod \"e972725e-489d-4f2a-a734-27d7eb570858\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " Oct 30 18:38:54.408303 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.408261 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e972725e-489d-4f2a-a734-27d7eb570858-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e972725e-489d-4f2a-a734-27d7eb570858-workload-socket\") pod \"e972725e-489d-4f2a-a734-27d7eb570858\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " Oct 30 18:38:54.408391 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.408366 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e972725e-489d-4f2a-a734-27d7eb570858-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e972725e-489d-4f2a-a734-27d7eb570858-istio-envoy\") pod \"e972725e-489d-4f2a-a734-27d7eb570858\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " Oct 30 18:38:54.408455 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.408405 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e972725e-489d-4f2a-a734-27d7eb570858-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e972725e-489d-4f2a-a734-27d7eb570858-workload-certs\") pod \"e972725e-489d-4f2a-a734-27d7eb570858\" (UID: \"e972725e-489d-4f2a-a734-27d7eb570858\") " Oct 30 18:38:54.409420 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.409393 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e972725e-489d-4f2a-a734-27d7eb570858-workload-certs" pod "e972725e-489d-4f2a-a734-27d7eb570858" (UID: "e972725e-489d-4f2a-a734-27d7eb570858"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.409654 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.409628 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e972725e-489d-4f2a-a734-27d7eb570858-credential-socket" pod "e972725e-489d-4f2a-a734-27d7eb570858" (UID: "e972725e-489d-4f2a-a734-27d7eb570858"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.409949 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.409926 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e972725e-489d-4f2a-a734-27d7eb570858-istiod-ca-cert" pod "e972725e-489d-4f2a-a734-27d7eb570858" (UID: "e972725e-489d-4f2a-a734-27d7eb570858"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:54.410196 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.410173 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e972725e-489d-4f2a-a734-27d7eb570858-istio-data" pod "e972725e-489d-4f2a-a734-27d7eb570858" (UID: "e972725e-489d-4f2a-a734-27d7eb570858"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.410418 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.410397 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e972725e-489d-4f2a-a734-27d7eb570858-workload-socket" pod "e972725e-489d-4f2a-a734-27d7eb570858" (UID: "e972725e-489d-4f2a-a734-27d7eb570858"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.417734 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.417702 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e972725e-489d-4f2a-a734-27d7eb570858-istio-envoy" pod "e972725e-489d-4f2a-a734-27d7eb570858" (UID: "e972725e-489d-4f2a-a734-27d7eb570858"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.418054 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.418008 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e972725e-489d-4f2a-a734-27d7eb570858-kube-api-access-r6sqd" pod "e972725e-489d-4f2a-a734-27d7eb570858" (UID: "e972725e-489d-4f2a-a734-27d7eb570858"). InnerVolumeSpecName "kube-api-access-r6sqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:54.419085 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.419053 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e972725e-489d-4f2a-a734-27d7eb570858-istio-token" pod "e972725e-489d-4f2a-a734-27d7eb570858" (UID: "e972725e-489d-4f2a-a734-27d7eb570858"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:54.421692 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.421663 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e972725e-489d-4f2a-a734-27d7eb570858-istio-podinfo" pod "e972725e-489d-4f2a-a734-27d7eb570858" (UID: "e972725e-489d-4f2a-a734-27d7eb570858"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:38:54.424432 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.424410 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" Oct 30 18:38:54.495035 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.494953 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" Oct 30 18:38:54.509945 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.509915 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac80ccea-a611-4de1-951c-479ad3d827a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac80ccea-a611-4de1-951c-479ad3d827a7-istio-envoy\") pod \"ac80ccea-a611-4de1-951c-479ad3d827a7\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " Oct 30 18:38:54.511367 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.511310 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac80ccea-a611-4de1-951c-479ad3d827a7-kube-api-access-5dknf\" (UniqueName: \"kubernetes.io/projected/ac80ccea-a611-4de1-951c-479ad3d827a7-kube-api-access-5dknf\") pod \"ac80ccea-a611-4de1-951c-479ad3d827a7\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " Oct 30 18:38:54.511584 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.511567 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac80ccea-a611-4de1-951c-479ad3d827a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac80ccea-a611-4de1-951c-479ad3d827a7-workload-socket\") pod \"ac80ccea-a611-4de1-951c-479ad3d827a7\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " Oct 30 18:38:54.511989 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.511878 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac80ccea-a611-4de1-951c-479ad3d827a7-workload-socket" pod "ac80ccea-a611-4de1-951c-479ad3d827a7" (UID: "ac80ccea-a611-4de1-951c-479ad3d827a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.512139 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.512114 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac80ccea-a611-4de1-951c-479ad3d827a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac80ccea-a611-4de1-951c-479ad3d827a7-istio-data\") pod \"ac80ccea-a611-4de1-951c-479ad3d827a7\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " Oct 30 18:38:54.512284 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.512188 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac80ccea-a611-4de1-951c-479ad3d827a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac80ccea-a611-4de1-951c-479ad3d827a7-workload-certs\") pod \"ac80ccea-a611-4de1-951c-479ad3d827a7\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " Oct 30 18:38:54.512390 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.512285 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac80ccea-a611-4de1-951c-479ad3d827a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac80ccea-a611-4de1-951c-479ad3d827a7-istio-podinfo\") pod \"ac80ccea-a611-4de1-951c-479ad3d827a7\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " Oct 30 18:38:54.512390 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.512351 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac80ccea-a611-4de1-951c-479ad3d827a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac80ccea-a611-4de1-951c-479ad3d827a7-istiod-ca-cert\") pod \"ac80ccea-a611-4de1-951c-479ad3d827a7\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " Oct 30 18:38:54.514067 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.512721 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac80ccea-a611-4de1-951c-479ad3d827a7-istio-envoy" pod "ac80ccea-a611-4de1-951c-479ad3d827a7" (UID: "ac80ccea-a611-4de1-951c-479ad3d827a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.514067 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.512612 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac80ccea-a611-4de1-951c-479ad3d827a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac80ccea-a611-4de1-951c-479ad3d827a7-credential-socket\") pod \"ac80ccea-a611-4de1-951c-479ad3d827a7\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " Oct 30 18:38:54.514067 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.512965 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac80ccea-a611-4de1-951c-479ad3d827a7-istio-data" pod "ac80ccea-a611-4de1-951c-479ad3d827a7" (UID: "ac80ccea-a611-4de1-951c-479ad3d827a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.514067 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.512991 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac80ccea-a611-4de1-951c-479ad3d827a7-istio-token\" (UniqueName: \"kubernetes.io/projected/ac80ccea-a611-4de1-951c-479ad3d827a7-istio-token\") pod \"ac80ccea-a611-4de1-951c-479ad3d827a7\" (UID: \"ac80ccea-a611-4de1-951c-479ad3d827a7\") " Oct 30 18:38:54.514067 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.513052 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac80ccea-a611-4de1-951c-479ad3d827a7-istiod-ca-cert" pod "ac80ccea-a611-4de1-951c-479ad3d827a7" (UID: "ac80ccea-a611-4de1-951c-479ad3d827a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:54.514067 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.513193 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac80ccea-a611-4de1-951c-479ad3d827a7-credential-socket" pod "ac80ccea-a611-4de1-951c-479ad3d827a7" (UID: "ac80ccea-a611-4de1-951c-479ad3d827a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.514067 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.513771 4455 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e972725e-489d-4f2a-a734-27d7eb570858-istiod-ca-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.514067 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.513794 4455 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e972725e-489d-4f2a-a734-27d7eb570858-istio-data\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.514067 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.513806 4455 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac80ccea-a611-4de1-951c-479ad3d827a7-istio-envoy\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.514067 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.513820 4455 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac80ccea-a611-4de1-951c-479ad3d827a7-workload-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.514067 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.513833 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6sqd\" (UniqueName: \"kubernetes.io/projected/e972725e-489d-4f2a-a734-27d7eb570858-kube-api-access-r6sqd\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.514067 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.513849 4455 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e972725e-489d-4f2a-a734-27d7eb570858-istio-podinfo\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.514067 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.513862 4455 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac80ccea-a611-4de1-951c-479ad3d827a7-istio-data\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.514067 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.513873 4455 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e972725e-489d-4f2a-a734-27d7eb570858-istio-token\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.514067 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.513885 4455 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac80ccea-a611-4de1-951c-479ad3d827a7-istiod-ca-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.514067 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.513898 4455 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e972725e-489d-4f2a-a734-27d7eb570858-workload-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.514067 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.513911 4455 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac80ccea-a611-4de1-951c-479ad3d827a7-credential-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.514067 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.513926 4455 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e972725e-489d-4f2a-a734-27d7eb570858-istio-envoy\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.514067 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.513938 4455 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e972725e-489d-4f2a-a734-27d7eb570858-workload-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.514067 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.513950 4455 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e972725e-489d-4f2a-a734-27d7eb570858-credential-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.514067 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.514025 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac80ccea-a611-4de1-951c-479ad3d827a7-workload-certs" pod "ac80ccea-a611-4de1-951c-479ad3d827a7" (UID: "ac80ccea-a611-4de1-951c-479ad3d827a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.516055 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.516015 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac80ccea-a611-4de1-951c-479ad3d827a7-istio-podinfo" pod "ac80ccea-a611-4de1-951c-479ad3d827a7" (UID: "ac80ccea-a611-4de1-951c-479ad3d827a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:38:54.516055 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.516044 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac80ccea-a611-4de1-951c-479ad3d827a7-kube-api-access-5dknf" pod "ac80ccea-a611-4de1-951c-479ad3d827a7" (UID: "ac80ccea-a611-4de1-951c-479ad3d827a7"). InnerVolumeSpecName "kube-api-access-5dknf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:54.517125 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.517087 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac80ccea-a611-4de1-951c-479ad3d827a7-istio-token" pod "ac80ccea-a611-4de1-951c-479ad3d827a7" (UID: "ac80ccea-a611-4de1-951c-479ad3d827a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:54.615735 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.615402 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93b710b6-b5a8-4fe0-ac70-861dd036b70d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b710b6-b5a8-4fe0-ac70-861dd036b70d-credential-socket\") pod \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " Oct 30 18:38:54.615735 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.615486 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93b710b6-b5a8-4fe0-ac70-861dd036b70d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b710b6-b5a8-4fe0-ac70-861dd036b70d-istio-podinfo\") pod \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " Oct 30 18:38:54.615735 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.615525 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93b710b6-b5a8-4fe0-ac70-861dd036b70d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b710b6-b5a8-4fe0-ac70-861dd036b70d-workload-socket\") pod \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " Oct 30 18:38:54.615735 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.615554 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93b710b6-b5a8-4fe0-ac70-861dd036b70d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b710b6-b5a8-4fe0-ac70-861dd036b70d-istio-envoy\") pod \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " Oct 30 18:38:54.615735 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.615707 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93b710b6-b5a8-4fe0-ac70-861dd036b70d-kube-api-access-6gjld\" (UniqueName: \"kubernetes.io/projected/93b710b6-b5a8-4fe0-ac70-861dd036b70d-kube-api-access-6gjld\") pod \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " Oct 30 18:38:54.616225 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.615761 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93b710b6-b5a8-4fe0-ac70-861dd036b70d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b710b6-b5a8-4fe0-ac70-861dd036b70d-istio-data\") pod \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " Oct 30 18:38:54.616225 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.615814 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93b710b6-b5a8-4fe0-ac70-861dd036b70d-istio-token\" (UniqueName: \"kubernetes.io/projected/93b710b6-b5a8-4fe0-ac70-861dd036b70d-istio-token\") pod \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " Oct 30 18:38:54.616225 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.615861 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93b710b6-b5a8-4fe0-ac70-861dd036b70d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b710b6-b5a8-4fe0-ac70-861dd036b70d-workload-certs\") pod \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " Oct 30 18:38:54.616225 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.615948 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93b710b6-b5a8-4fe0-ac70-861dd036b70d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b710b6-b5a8-4fe0-ac70-861dd036b70d-istiod-ca-cert\") pod \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\" (UID: \"93b710b6-b5a8-4fe0-ac70-861dd036b70d\") " Oct 30 18:38:54.616225 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.616061 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b710b6-b5a8-4fe0-ac70-861dd036b70d-workload-socket" pod "93b710b6-b5a8-4fe0-ac70-861dd036b70d" (UID: "93b710b6-b5a8-4fe0-ac70-861dd036b70d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.616539 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.616302 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b710b6-b5a8-4fe0-ac70-861dd036b70d-credential-socket" pod "93b710b6-b5a8-4fe0-ac70-861dd036b70d" (UID: "93b710b6-b5a8-4fe0-ac70-861dd036b70d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.616716 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.616682 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93b710b6-b5a8-4fe0-ac70-861dd036b70d-istiod-ca-cert" pod "93b710b6-b5a8-4fe0-ac70-861dd036b70d" (UID: "93b710b6-b5a8-4fe0-ac70-861dd036b70d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:54.616852 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.616809 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b710b6-b5a8-4fe0-ac70-861dd036b70d-istio-data" pod "93b710b6-b5a8-4fe0-ac70-861dd036b70d" (UID: "93b710b6-b5a8-4fe0-ac70-861dd036b70d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.616941 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.616920 4455 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac80ccea-a611-4de1-951c-479ad3d827a7-workload-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.617011 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.616947 4455 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b710b6-b5a8-4fe0-ac70-861dd036b70d-credential-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.617011 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.616961 4455 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b710b6-b5a8-4fe0-ac70-861dd036b70d-workload-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.617011 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.616974 4455 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac80ccea-a611-4de1-951c-479ad3d827a7-istio-podinfo\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.617011 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.616988 4455 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac80ccea-a611-4de1-951c-479ad3d827a7-istio-token\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.617011 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.617002 4455 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b710b6-b5a8-4fe0-ac70-861dd036b70d-istiod-ca-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.617366 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.617015 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dknf\" (UniqueName: \"kubernetes.io/projected/ac80ccea-a611-4de1-951c-479ad3d827a7-kube-api-access-5dknf\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.617771 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.617748 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b710b6-b5a8-4fe0-ac70-861dd036b70d-workload-certs" pod "93b710b6-b5a8-4fe0-ac70-861dd036b70d" (UID: "93b710b6-b5a8-4fe0-ac70-861dd036b70d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.619662 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.619636 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b710b6-b5a8-4fe0-ac70-861dd036b70d-kube-api-access-6gjld" pod "93b710b6-b5a8-4fe0-ac70-861dd036b70d" (UID: "93b710b6-b5a8-4fe0-ac70-861dd036b70d"). InnerVolumeSpecName "kube-api-access-6gjld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:54.619743 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.619727 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93b710b6-b5a8-4fe0-ac70-861dd036b70d-istio-podinfo" pod "93b710b6-b5a8-4fe0-ac70-861dd036b70d" (UID: "93b710b6-b5a8-4fe0-ac70-861dd036b70d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:38:54.619786 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.619739 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b710b6-b5a8-4fe0-ac70-861dd036b70d-istio-envoy" pod "93b710b6-b5a8-4fe0-ac70-861dd036b70d" (UID: "93b710b6-b5a8-4fe0-ac70-861dd036b70d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.620416 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.620394 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b710b6-b5a8-4fe0-ac70-861dd036b70d-istio-token" pod "93b710b6-b5a8-4fe0-ac70-861dd036b70d" (UID: "93b710b6-b5a8-4fe0-ac70-861dd036b70d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:54.719189 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.719158 4455 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b710b6-b5a8-4fe0-ac70-861dd036b70d-istio-podinfo\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.719189 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.719188 4455 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b710b6-b5a8-4fe0-ac70-861dd036b70d-istio-envoy\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.719446 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.719204 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gjld\" (UniqueName: \"kubernetes.io/projected/93b710b6-b5a8-4fe0-ac70-861dd036b70d-kube-api-access-6gjld\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.719446 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.719216 4455 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b710b6-b5a8-4fe0-ac70-861dd036b70d-istio-data\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.719446 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.719228 4455 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b710b6-b5a8-4fe0-ac70-861dd036b70d-istio-token\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.719446 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.719240 4455 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b710b6-b5a8-4fe0-ac70-861dd036b70d-workload-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.952184 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.952071 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" Oct 30 18:38:54.952184 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.952160 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" event={"ID":"e972725e-489d-4f2a-a734-27d7eb570858","Type":"ContainerDied","Data":"946f14782c693823bc55858538440fc8959065207e645894ca6d65a811047082"} Oct 30 18:38:54.952471 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.952239 4455 scope.go:122] "RemoveContainer" containerID="946f14782c693823bc55858538440fc8959065207e645894ca6d65a811047082" Oct 30 18:38:54.955733 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.951946 4455 generic.go:356] "Generic (PLEG): container finished" podID="e972725e-489d-4f2a-a734-27d7eb570858" containerID="946f14782c693823bc55858538440fc8959065207e645894ca6d65a811047082" exitCode=0 Oct 30 18:38:54.955733 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.955665 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x" event={"ID":"e972725e-489d-4f2a-a734-27d7eb570858","Type":"ContainerDied","Data":"8b87759888df73f134a91e5308819b7deee84f441e13d2d98cbc4b25ea3ecba9"} Oct 30 18:38:54.965366 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.962177 4455 generic.go:356] "Generic (PLEG): container finished" podID="ac80ccea-a611-4de1-951c-479ad3d827a7" containerID="36e7d4a4bb7018d8514f41830ca5589fbd87445555b98301f09584b942a62f0f" exitCode=0 Oct 30 18:38:54.965366 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.962344 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" event={"ID":"ac80ccea-a611-4de1-951c-479ad3d827a7","Type":"ContainerDied","Data":"36e7d4a4bb7018d8514f41830ca5589fbd87445555b98301f09584b942a62f0f"} Oct 30 18:38:54.965366 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.962402 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" event={"ID":"ac80ccea-a611-4de1-951c-479ad3d827a7","Type":"ContainerDied","Data":"06b57ac3c4ad611959006b0fe39972cde4f9b5382652d440f5ecb84b6d98ba8e"} Oct 30 18:38:54.965366 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.962490 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp" Oct 30 18:38:54.976172 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.967988 4455 generic.go:356] "Generic (PLEG): container finished" podID="93b710b6-b5a8-4fe0-ac70-861dd036b70d" containerID="b693433aac4f98920aa85577a8ea16b04a62c0737ed7789f243a93630cc5955d" exitCode=0 Oct 30 18:38:54.976172 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.968144 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" event={"ID":"93b710b6-b5a8-4fe0-ac70-861dd036b70d","Type":"ContainerDied","Data":"b693433aac4f98920aa85577a8ea16b04a62c0737ed7789f243a93630cc5955d"} Oct 30 18:38:54.976172 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.968195 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" event={"ID":"93b710b6-b5a8-4fe0-ac70-861dd036b70d","Type":"ContainerDied","Data":"aaa60f6ed92f62b1ac953df682edf79e8151a786a420e01cac0cfe72ee1a9d41"} Oct 30 18:38:54.976172 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.968281 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26" Oct 30 18:38:54.984712 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.984197 4455 scope.go:122] "RemoveContainer" containerID="946f14782c693823bc55858538440fc8959065207e645894ca6d65a811047082" Oct 30 18:38:54.984712 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:38:54.984664 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"946f14782c693823bc55858538440fc8959065207e645894ca6d65a811047082\": container with ID starting with 946f14782c693823bc55858538440fc8959065207e645894ca6d65a811047082 not found: ID does not exist" containerID="946f14782c693823bc55858538440fc8959065207e645894ca6d65a811047082" Oct 30 18:38:54.984712 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.984693 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"946f14782c693823bc55858538440fc8959065207e645894ca6d65a811047082"} err="failed to get container status \"946f14782c693823bc55858538440fc8959065207e645894ca6d65a811047082\": rpc error: code = NotFound desc = could not find container \"946f14782c693823bc55858538440fc8959065207e645894ca6d65a811047082\": container with ID starting with 946f14782c693823bc55858538440fc8959065207e645894ca6d65a811047082 not found: ID does not exist" Oct 30 18:38:54.984712 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:54.984710 4455 scope.go:122] "RemoveContainer" containerID="36e7d4a4bb7018d8514f41830ca5589fbd87445555b98301f09584b942a62f0f" Oct 30 18:38:55.026526 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:55.026506 4455 scope.go:122] "RemoveContainer" containerID="36e7d4a4bb7018d8514f41830ca5589fbd87445555b98301f09584b942a62f0f" Oct 30 18:38:55.026861 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:38:55.026840 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36e7d4a4bb7018d8514f41830ca5589fbd87445555b98301f09584b942a62f0f\": container with ID starting with 36e7d4a4bb7018d8514f41830ca5589fbd87445555b98301f09584b942a62f0f not found: ID does not exist" containerID="36e7d4a4bb7018d8514f41830ca5589fbd87445555b98301f09584b942a62f0f" Oct 30 18:38:55.026959 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:55.026866 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36e7d4a4bb7018d8514f41830ca5589fbd87445555b98301f09584b942a62f0f"} err="failed to get container status \"36e7d4a4bb7018d8514f41830ca5589fbd87445555b98301f09584b942a62f0f\": rpc error: code = NotFound desc = could not find container \"36e7d4a4bb7018d8514f41830ca5589fbd87445555b98301f09584b942a62f0f\": container with ID starting with 36e7d4a4bb7018d8514f41830ca5589fbd87445555b98301f09584b942a62f0f not found: ID does not exist" Oct 30 18:38:55.026959 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:55.026883 4455 scope.go:122] "RemoveContainer" containerID="b693433aac4f98920aa85577a8ea16b04a62c0737ed7789f243a93630cc5955d" Oct 30 18:38:55.036715 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:55.036690 4455 scope.go:122] "RemoveContainer" containerID="b693433aac4f98920aa85577a8ea16b04a62c0737ed7789f243a93630cc5955d" Oct 30 18:38:55.036984 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:38:55.036965 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b693433aac4f98920aa85577a8ea16b04a62c0737ed7789f243a93630cc5955d\": container with ID starting with b693433aac4f98920aa85577a8ea16b04a62c0737ed7789f243a93630cc5955d not found: ID does not exist" containerID="b693433aac4f98920aa85577a8ea16b04a62c0737ed7789f243a93630cc5955d" Oct 30 18:38:55.037046 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:55.036994 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b693433aac4f98920aa85577a8ea16b04a62c0737ed7789f243a93630cc5955d"} err="failed to get container status \"b693433aac4f98920aa85577a8ea16b04a62c0737ed7789f243a93630cc5955d\": rpc error: code = NotFound desc = could not find container \"b693433aac4f98920aa85577a8ea16b04a62c0737ed7789f243a93630cc5955d\": container with ID starting with b693433aac4f98920aa85577a8ea16b04a62c0737ed7789f243a93630cc5955d not found: ID does not exist" Oct 30 18:38:55.072586 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:55.072554 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x"] Oct 30 18:38:55.076892 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:55.076864 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-c9p6x"] Oct 30 18:38:55.118130 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:55.118103 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26"] Oct 30 18:38:55.130895 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:55.130867 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-k9g26"] Oct 30 18:38:55.186198 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:55.186161 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp"] Oct 30 18:38:55.191466 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:55.191440 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-r5mjp"] Oct 30 18:38:55.635437 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:55.635399 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93b710b6-b5a8-4fe0-ac70-861dd036b70d" path="/var/lib/kubelet/pods/93b710b6-b5a8-4fe0-ac70-861dd036b70d/volumes" Oct 30 18:38:55.636116 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:55.636091 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac80ccea-a611-4de1-951c-479ad3d827a7" path="/var/lib/kubelet/pods/ac80ccea-a611-4de1-951c-479ad3d827a7/volumes" Oct 30 18:38:55.636695 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:55.636672 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e972725e-489d-4f2a-a734-27d7eb570858" path="/var/lib/kubelet/pods/e972725e-489d-4f2a-a734-27d7eb570858/volumes" Oct 30 18:38:57.465106 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.464719 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" Oct 30 18:38:57.497818 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.497792 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" Oct 30 18:38:57.530155 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.530115 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c9b7598-cec3-43ab-90cf-93e7b407d68e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c9b7598-cec3-43ab-90cf-93e7b407d68e-istio-podinfo\") pod \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " Oct 30 18:38:57.530305 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.530174 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c9b7598-cec3-43ab-90cf-93e7b407d68e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9b7598-cec3-43ab-90cf-93e7b407d68e-workload-socket\") pod \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " Oct 30 18:38:57.530305 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.530219 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c9b7598-cec3-43ab-90cf-93e7b407d68e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9b7598-cec3-43ab-90cf-93e7b407d68e-credential-socket\") pod \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " Oct 30 18:38:57.530305 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.530264 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3daa8589-9def-486f-82ee-0c923a18b68b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daa8589-9def-486f-82ee-0c923a18b68b-workload-socket\") pod \"3daa8589-9def-486f-82ee-0c923a18b68b\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " Oct 30 18:38:57.530488 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.530342 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3daa8589-9def-486f-82ee-0c923a18b68b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3daa8589-9def-486f-82ee-0c923a18b68b-istio-podinfo\") pod \"3daa8589-9def-486f-82ee-0c923a18b68b\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " Oct 30 18:38:57.530488 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.530401 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3daa8589-9def-486f-82ee-0c923a18b68b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3daa8589-9def-486f-82ee-0c923a18b68b-istio-data\") pod \"3daa8589-9def-486f-82ee-0c923a18b68b\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " Oct 30 18:38:57.530593 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.530488 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3daa8589-9def-486f-82ee-0c923a18b68b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3daa8589-9def-486f-82ee-0c923a18b68b-istiod-ca-cert\") pod \"3daa8589-9def-486f-82ee-0c923a18b68b\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " Oct 30 18:38:57.530593 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.530525 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c9b7598-cec3-43ab-90cf-93e7b407d68e-workload-socket" pod "7c9b7598-cec3-43ab-90cf-93e7b407d68e" (UID: "7c9b7598-cec3-43ab-90cf-93e7b407d68e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.530593 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.530538 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c9b7598-cec3-43ab-90cf-93e7b407d68e-kube-api-access-5vx4g\" (UniqueName: \"kubernetes.io/projected/7c9b7598-cec3-43ab-90cf-93e7b407d68e-kube-api-access-5vx4g\") pod \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " Oct 30 18:38:57.530593 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.530565 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c9b7598-cec3-43ab-90cf-93e7b407d68e-credential-socket" pod "7c9b7598-cec3-43ab-90cf-93e7b407d68e" (UID: "7c9b7598-cec3-43ab-90cf-93e7b407d68e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.530817 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.530639 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c9b7598-cec3-43ab-90cf-93e7b407d68e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c9b7598-cec3-43ab-90cf-93e7b407d68e-istio-envoy\") pod \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " Oct 30 18:38:57.530817 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.530718 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3daa8589-9def-486f-82ee-0c923a18b68b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3daa8589-9def-486f-82ee-0c923a18b68b-istio-envoy\") pod \"3daa8589-9def-486f-82ee-0c923a18b68b\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " Oct 30 18:38:57.530817 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.530733 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3daa8589-9def-486f-82ee-0c923a18b68b-istio-data" pod "3daa8589-9def-486f-82ee-0c923a18b68b" (UID: "3daa8589-9def-486f-82ee-0c923a18b68b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.530817 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.530772 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3daa8589-9def-486f-82ee-0c923a18b68b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3daa8589-9def-486f-82ee-0c923a18b68b-workload-certs\") pod \"3daa8589-9def-486f-82ee-0c923a18b68b\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " Oct 30 18:38:57.530817 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.530803 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3daa8589-9def-486f-82ee-0c923a18b68b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daa8589-9def-486f-82ee-0c923a18b68b-credential-socket\") pod \"3daa8589-9def-486f-82ee-0c923a18b68b\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " Oct 30 18:38:57.531091 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.530841 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3daa8589-9def-486f-82ee-0c923a18b68b-istio-token\" (UniqueName: \"kubernetes.io/projected/3daa8589-9def-486f-82ee-0c923a18b68b-istio-token\") pod \"3daa8589-9def-486f-82ee-0c923a18b68b\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " Oct 30 18:38:57.531091 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.530879 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3daa8589-9def-486f-82ee-0c923a18b68b-workload-socket" pod "3daa8589-9def-486f-82ee-0c923a18b68b" (UID: "3daa8589-9def-486f-82ee-0c923a18b68b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.531091 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.530930 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c9b7598-cec3-43ab-90cf-93e7b407d68e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c9b7598-cec3-43ab-90cf-93e7b407d68e-istio-data\") pod \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " Oct 30 18:38:57.531091 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.530943 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3daa8589-9def-486f-82ee-0c923a18b68b-istiod-ca-cert" pod "3daa8589-9def-486f-82ee-0c923a18b68b" (UID: "3daa8589-9def-486f-82ee-0c923a18b68b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:57.531091 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.530965 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3daa8589-9def-486f-82ee-0c923a18b68b-kube-api-access-zndlf\" (UniqueName: \"kubernetes.io/projected/3daa8589-9def-486f-82ee-0c923a18b68b-kube-api-access-zndlf\") pod \"3daa8589-9def-486f-82ee-0c923a18b68b\" (UID: \"3daa8589-9def-486f-82ee-0c923a18b68b\") " Oct 30 18:38:57.531091 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.531022 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c9b7598-cec3-43ab-90cf-93e7b407d68e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c9b7598-cec3-43ab-90cf-93e7b407d68e-istiod-ca-cert\") pod \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " Oct 30 18:38:57.531091 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.531055 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c9b7598-cec3-43ab-90cf-93e7b407d68e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c9b7598-cec3-43ab-90cf-93e7b407d68e-workload-certs\") pod \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " Oct 30 18:38:57.531494 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.531105 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3daa8589-9def-486f-82ee-0c923a18b68b-workload-certs" pod "3daa8589-9def-486f-82ee-0c923a18b68b" (UID: "3daa8589-9def-486f-82ee-0c923a18b68b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.531494 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.531199 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c9b7598-cec3-43ab-90cf-93e7b407d68e-istio-token\" (UniqueName: \"kubernetes.io/projected/7c9b7598-cec3-43ab-90cf-93e7b407d68e-istio-token\") pod \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\" (UID: \"7c9b7598-cec3-43ab-90cf-93e7b407d68e\") " Oct 30 18:38:57.531494 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.531211 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" Oct 30 18:38:57.531494 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.531316 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c9b7598-cec3-43ab-90cf-93e7b407d68e-workload-certs" pod "7c9b7598-cec3-43ab-90cf-93e7b407d68e" (UID: "7c9b7598-cec3-43ab-90cf-93e7b407d68e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.531808 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.531788 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c9b7598-cec3-43ab-90cf-93e7b407d68e-istio-data" pod "7c9b7598-cec3-43ab-90cf-93e7b407d68e" (UID: "7c9b7598-cec3-43ab-90cf-93e7b407d68e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.531895 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.531841 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c9b7598-cec3-43ab-90cf-93e7b407d68e-istiod-ca-cert" pod "7c9b7598-cec3-43ab-90cf-93e7b407d68e" (UID: "7c9b7598-cec3-43ab-90cf-93e7b407d68e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:57.532108 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.532080 4455 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9b7598-cec3-43ab-90cf-93e7b407d68e-workload-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.532199 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.532116 4455 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9b7598-cec3-43ab-90cf-93e7b407d68e-credential-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.532199 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.532133 4455 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daa8589-9def-486f-82ee-0c923a18b68b-workload-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.532199 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.532147 4455 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3daa8589-9def-486f-82ee-0c923a18b68b-istio-data\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.532199 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.532161 4455 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3daa8589-9def-486f-82ee-0c923a18b68b-istiod-ca-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.532199 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.532174 4455 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3daa8589-9def-486f-82ee-0c923a18b68b-workload-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.532199 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.532187 4455 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c9b7598-cec3-43ab-90cf-93e7b407d68e-istio-data\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.532199 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.532200 4455 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c9b7598-cec3-43ab-90cf-93e7b407d68e-istiod-ca-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.532199 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.532215 4455 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c9b7598-cec3-43ab-90cf-93e7b407d68e-workload-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.532703 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.532663 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3daa8589-9def-486f-82ee-0c923a18b68b-credential-socket" pod "3daa8589-9def-486f-82ee-0c923a18b68b" (UID: "3daa8589-9def-486f-82ee-0c923a18b68b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.534487 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.534462 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3daa8589-9def-486f-82ee-0c923a18b68b-istio-token" pod "3daa8589-9def-486f-82ee-0c923a18b68b" (UID: "3daa8589-9def-486f-82ee-0c923a18b68b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:57.536468 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.536445 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c9b7598-cec3-43ab-90cf-93e7b407d68e-istio-token" pod "7c9b7598-cec3-43ab-90cf-93e7b407d68e" (UID: "7c9b7598-cec3-43ab-90cf-93e7b407d68e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:57.536805 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.536785 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3daa8589-9def-486f-82ee-0c923a18b68b-kube-api-access-zndlf" pod "3daa8589-9def-486f-82ee-0c923a18b68b" (UID: "3daa8589-9def-486f-82ee-0c923a18b68b"). InnerVolumeSpecName "kube-api-access-zndlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:57.537572 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.537549 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3daa8589-9def-486f-82ee-0c923a18b68b-istio-podinfo" pod "3daa8589-9def-486f-82ee-0c923a18b68b" (UID: "3daa8589-9def-486f-82ee-0c923a18b68b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:38:57.537877 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.537606 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c9b7598-cec3-43ab-90cf-93e7b407d68e-kube-api-access-5vx4g" pod "7c9b7598-cec3-43ab-90cf-93e7b407d68e" (UID: "7c9b7598-cec3-43ab-90cf-93e7b407d68e"). InnerVolumeSpecName "kube-api-access-5vx4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:57.537877 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.537678 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c9b7598-cec3-43ab-90cf-93e7b407d68e-istio-podinfo" pod "7c9b7598-cec3-43ab-90cf-93e7b407d68e" (UID: "7c9b7598-cec3-43ab-90cf-93e7b407d68e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:38:57.539483 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.539456 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3daa8589-9def-486f-82ee-0c923a18b68b-istio-envoy" pod "3daa8589-9def-486f-82ee-0c923a18b68b" (UID: "3daa8589-9def-486f-82ee-0c923a18b68b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.541518 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.541483 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c9b7598-cec3-43ab-90cf-93e7b407d68e-istio-envoy" pod "7c9b7598-cec3-43ab-90cf-93e7b407d68e" (UID: "7c9b7598-cec3-43ab-90cf-93e7b407d68e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.632713 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.632612 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbf4bc50-3670-40de-8159-3d8805fdd601-kube-api-access-d8w5p\" (UniqueName: \"kubernetes.io/projected/bbf4bc50-3670-40de-8159-3d8805fdd601-kube-api-access-d8w5p\") pod \"bbf4bc50-3670-40de-8159-3d8805fdd601\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " Oct 30 18:38:57.632713 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.632680 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbf4bc50-3670-40de-8159-3d8805fdd601-istio-token\" (UniqueName: \"kubernetes.io/projected/bbf4bc50-3670-40de-8159-3d8805fdd601-istio-token\") pod \"bbf4bc50-3670-40de-8159-3d8805fdd601\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " Oct 30 18:38:57.632969 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.632839 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbf4bc50-3670-40de-8159-3d8805fdd601-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf4bc50-3670-40de-8159-3d8805fdd601-istio-envoy\") pod \"bbf4bc50-3670-40de-8159-3d8805fdd601\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " Oct 30 18:38:57.632969 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.632913 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbf4bc50-3670-40de-8159-3d8805fdd601-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf4bc50-3670-40de-8159-3d8805fdd601-istio-data\") pod \"bbf4bc50-3670-40de-8159-3d8805fdd601\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " Oct 30 18:38:57.633093 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.632998 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbf4bc50-3670-40de-8159-3d8805fdd601-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf4bc50-3670-40de-8159-3d8805fdd601-istio-podinfo\") pod \"bbf4bc50-3670-40de-8159-3d8805fdd601\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " Oct 30 18:38:57.633093 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.633026 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbf4bc50-3670-40de-8159-3d8805fdd601-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf4bc50-3670-40de-8159-3d8805fdd601-workload-socket\") pod \"bbf4bc50-3670-40de-8159-3d8805fdd601\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " Oct 30 18:38:57.633093 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.633062 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbf4bc50-3670-40de-8159-3d8805fdd601-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf4bc50-3670-40de-8159-3d8805fdd601-workload-certs\") pod \"bbf4bc50-3670-40de-8159-3d8805fdd601\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " Oct 30 18:38:57.633269 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.633161 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbf4bc50-3670-40de-8159-3d8805fdd601-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf4bc50-3670-40de-8159-3d8805fdd601-credential-socket\") pod \"bbf4bc50-3670-40de-8159-3d8805fdd601\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " Oct 30 18:38:57.633269 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.633193 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbf4bc50-3670-40de-8159-3d8805fdd601-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf4bc50-3670-40de-8159-3d8805fdd601-istiod-ca-cert\") pod \"bbf4bc50-3670-40de-8159-3d8805fdd601\" (UID: \"bbf4bc50-3670-40de-8159-3d8805fdd601\") " Oct 30 18:38:57.633269 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.633194 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf4bc50-3670-40de-8159-3d8805fdd601-istio-data" pod "bbf4bc50-3670-40de-8159-3d8805fdd601" (UID: "bbf4bc50-3670-40de-8159-3d8805fdd601"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.633466 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.633414 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf4bc50-3670-40de-8159-3d8805fdd601-workload-socket" pod "bbf4bc50-3670-40de-8159-3d8805fdd601" (UID: "bbf4bc50-3670-40de-8159-3d8805fdd601"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.633760 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.633737 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zndlf\" (UniqueName: \"kubernetes.io/projected/3daa8589-9def-486f-82ee-0c923a18b68b-kube-api-access-zndlf\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.633853 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.633768 4455 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c9b7598-cec3-43ab-90cf-93e7b407d68e-istio-token\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.633853 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.633782 4455 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c9b7598-cec3-43ab-90cf-93e7b407d68e-istio-podinfo\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.633853 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.633794 4455 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf4bc50-3670-40de-8159-3d8805fdd601-istio-data\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.633853 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.633807 4455 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3daa8589-9def-486f-82ee-0c923a18b68b-istio-podinfo\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.633853 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.633819 4455 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf4bc50-3670-40de-8159-3d8805fdd601-workload-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.633853 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.633832 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vx4g\" (UniqueName: \"kubernetes.io/projected/7c9b7598-cec3-43ab-90cf-93e7b407d68e-kube-api-access-5vx4g\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.633853 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.633847 4455 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c9b7598-cec3-43ab-90cf-93e7b407d68e-istio-envoy\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.634315 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.633861 4455 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3daa8589-9def-486f-82ee-0c923a18b68b-istio-envoy\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.634315 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.633873 4455 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daa8589-9def-486f-82ee-0c923a18b68b-credential-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.634315 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.633886 4455 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3daa8589-9def-486f-82ee-0c923a18b68b-istio-token\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.634315 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.634087 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf4bc50-3670-40de-8159-3d8805fdd601-workload-certs" pod "bbf4bc50-3670-40de-8159-3d8805fdd601" (UID: "bbf4bc50-3670-40de-8159-3d8805fdd601"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.634315 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.634252 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf4bc50-3670-40de-8159-3d8805fdd601-credential-socket" pod "bbf4bc50-3670-40de-8159-3d8805fdd601" (UID: "bbf4bc50-3670-40de-8159-3d8805fdd601"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.634616 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.634527 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbf4bc50-3670-40de-8159-3d8805fdd601-istiod-ca-cert" pod "bbf4bc50-3670-40de-8159-3d8805fdd601" (UID: "bbf4bc50-3670-40de-8159-3d8805fdd601"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:57.635527 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.635496 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf4bc50-3670-40de-8159-3d8805fdd601-kube-api-access-d8w5p" pod "bbf4bc50-3670-40de-8159-3d8805fdd601" (UID: "bbf4bc50-3670-40de-8159-3d8805fdd601"). InnerVolumeSpecName "kube-api-access-d8w5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:57.635907 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.635881 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf4bc50-3670-40de-8159-3d8805fdd601-istio-envoy" pod "bbf4bc50-3670-40de-8159-3d8805fdd601" (UID: "bbf4bc50-3670-40de-8159-3d8805fdd601"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.637961 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.637934 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf4bc50-3670-40de-8159-3d8805fdd601-istio-token" pod "bbf4bc50-3670-40de-8159-3d8805fdd601" (UID: "bbf4bc50-3670-40de-8159-3d8805fdd601"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:57.638720 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.638696 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbf4bc50-3670-40de-8159-3d8805fdd601-istio-podinfo" pod "bbf4bc50-3670-40de-8159-3d8805fdd601" (UID: "bbf4bc50-3670-40de-8159-3d8805fdd601"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:38:57.735614 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.735570 4455 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf4bc50-3670-40de-8159-3d8805fdd601-istio-podinfo\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.735614 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.735604 4455 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf4bc50-3670-40de-8159-3d8805fdd601-workload-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.735614 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.735620 4455 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf4bc50-3670-40de-8159-3d8805fdd601-credential-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.735864 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.735638 4455 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf4bc50-3670-40de-8159-3d8805fdd601-istiod-ca-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.735864 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.735653 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8w5p\" (UniqueName: \"kubernetes.io/projected/bbf4bc50-3670-40de-8159-3d8805fdd601-kube-api-access-d8w5p\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.735864 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.735668 4455 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbf4bc50-3670-40de-8159-3d8805fdd601-istio-token\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.735864 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.735684 4455 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf4bc50-3670-40de-8159-3d8805fdd601-istio-envoy\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.998819 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.998767 4455 generic.go:356] "Generic (PLEG): container finished" podID="bbf4bc50-3670-40de-8159-3d8805fdd601" containerID="f7d22f95cf4835ebe39cfc8659ce9bf4786d0f2fa16aafda6f92cbf7fcc80cac" exitCode=0 Oct 30 18:38:57.999039 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.998850 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" Oct 30 18:38:57.999039 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.998871 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" event={"ID":"bbf4bc50-3670-40de-8159-3d8805fdd601","Type":"ContainerDied","Data":"f7d22f95cf4835ebe39cfc8659ce9bf4786d0f2fa16aafda6f92cbf7fcc80cac"} Oct 30 18:38:57.999039 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.998943 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98" event={"ID":"bbf4bc50-3670-40de-8159-3d8805fdd601","Type":"ContainerDied","Data":"e8ecf9856e887677126de03fdf61388c8a4e29d8daad7c96fa5ad30a3d09f977"} Oct 30 18:38:57.999039 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:57.998974 4455 scope.go:122] "RemoveContainer" containerID="f7d22f95cf4835ebe39cfc8659ce9bf4786d0f2fa16aafda6f92cbf7fcc80cac" Oct 30 18:38:58.001957 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:58.001550 4455 generic.go:356] "Generic (PLEG): container finished" podID="3daa8589-9def-486f-82ee-0c923a18b68b" containerID="9736a79721eca5b603182cff32bfc216e3c642b536bb8178a0f7d54b8017d80f" exitCode=0 Oct 30 18:38:58.001957 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:58.001712 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" event={"ID":"3daa8589-9def-486f-82ee-0c923a18b68b","Type":"ContainerDied","Data":"9736a79721eca5b603182cff32bfc216e3c642b536bb8178a0f7d54b8017d80f"} Oct 30 18:38:58.001957 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:58.001755 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" event={"ID":"3daa8589-9def-486f-82ee-0c923a18b68b","Type":"ContainerDied","Data":"ad191e98f17f00ba8e5a4ab55853ec4faf59e0449ca435b8fb612d0c1394a7d8"} Oct 30 18:38:58.001957 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:58.001822 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf" Oct 30 18:38:58.004969 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:58.004941 4455 generic.go:356] "Generic (PLEG): container finished" podID="7c9b7598-cec3-43ab-90cf-93e7b407d68e" containerID="524f212977033a1bd2de88b5b493663bfe5755c1ed253ef4a3df7b5be1378a5b" exitCode=0 Oct 30 18:38:58.005115 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:58.004991 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" event={"ID":"7c9b7598-cec3-43ab-90cf-93e7b407d68e","Type":"ContainerDied","Data":"524f212977033a1bd2de88b5b493663bfe5755c1ed253ef4a3df7b5be1378a5b"} Oct 30 18:38:58.005115 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:58.004999 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" Oct 30 18:38:58.005115 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:58.005030 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j" event={"ID":"7c9b7598-cec3-43ab-90cf-93e7b407d68e","Type":"ContainerDied","Data":"828081dbe58ed059499f4ed94440685144eb4355c6f6d16773c6a1d37481b2d6"} Oct 30 18:38:58.016928 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:58.016905 4455 scope.go:122] "RemoveContainer" containerID="f7d22f95cf4835ebe39cfc8659ce9bf4786d0f2fa16aafda6f92cbf7fcc80cac" Oct 30 18:38:58.017259 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:38:58.017235 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7d22f95cf4835ebe39cfc8659ce9bf4786d0f2fa16aafda6f92cbf7fcc80cac\": container with ID starting with f7d22f95cf4835ebe39cfc8659ce9bf4786d0f2fa16aafda6f92cbf7fcc80cac not found: ID does not exist" containerID="f7d22f95cf4835ebe39cfc8659ce9bf4786d0f2fa16aafda6f92cbf7fcc80cac" Oct 30 18:38:58.017419 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:58.017264 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7d22f95cf4835ebe39cfc8659ce9bf4786d0f2fa16aafda6f92cbf7fcc80cac"} err="failed to get container status \"f7d22f95cf4835ebe39cfc8659ce9bf4786d0f2fa16aafda6f92cbf7fcc80cac\": rpc error: code = NotFound desc = could not find container \"f7d22f95cf4835ebe39cfc8659ce9bf4786d0f2fa16aafda6f92cbf7fcc80cac\": container with ID starting with f7d22f95cf4835ebe39cfc8659ce9bf4786d0f2fa16aafda6f92cbf7fcc80cac not found: ID does not exist" Oct 30 18:38:58.017419 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:58.017281 4455 scope.go:122] "RemoveContainer" containerID="9736a79721eca5b603182cff32bfc216e3c642b536bb8178a0f7d54b8017d80f" Oct 30 18:38:58.031064 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:58.031042 4455 scope.go:122] "RemoveContainer" containerID="9736a79721eca5b603182cff32bfc216e3c642b536bb8178a0f7d54b8017d80f" Oct 30 18:38:58.031387 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:38:58.031366 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9736a79721eca5b603182cff32bfc216e3c642b536bb8178a0f7d54b8017d80f\": container with ID starting with 9736a79721eca5b603182cff32bfc216e3c642b536bb8178a0f7d54b8017d80f not found: ID does not exist" containerID="9736a79721eca5b603182cff32bfc216e3c642b536bb8178a0f7d54b8017d80f" Oct 30 18:38:58.031440 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:58.031397 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9736a79721eca5b603182cff32bfc216e3c642b536bb8178a0f7d54b8017d80f"} err="failed to get container status \"9736a79721eca5b603182cff32bfc216e3c642b536bb8178a0f7d54b8017d80f\": rpc error: code = NotFound desc = could not find container \"9736a79721eca5b603182cff32bfc216e3c642b536bb8178a0f7d54b8017d80f\": container with ID starting with 9736a79721eca5b603182cff32bfc216e3c642b536bb8178a0f7d54b8017d80f not found: ID does not exist" Oct 30 18:38:58.031440 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:58.031414 4455 scope.go:122] "RemoveContainer" containerID="524f212977033a1bd2de88b5b493663bfe5755c1ed253ef4a3df7b5be1378a5b" Oct 30 18:38:58.040927 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:58.040911 4455 scope.go:122] "RemoveContainer" containerID="524f212977033a1bd2de88b5b493663bfe5755c1ed253ef4a3df7b5be1378a5b" Oct 30 18:38:58.041193 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:38:58.041174 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"524f212977033a1bd2de88b5b493663bfe5755c1ed253ef4a3df7b5be1378a5b\": container with ID starting with 524f212977033a1bd2de88b5b493663bfe5755c1ed253ef4a3df7b5be1378a5b not found: ID does not exist" containerID="524f212977033a1bd2de88b5b493663bfe5755c1ed253ef4a3df7b5be1378a5b" Oct 30 18:38:58.041239 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:58.041201 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"524f212977033a1bd2de88b5b493663bfe5755c1ed253ef4a3df7b5be1378a5b"} err="failed to get container status \"524f212977033a1bd2de88b5b493663bfe5755c1ed253ef4a3df7b5be1378a5b\": rpc error: code = NotFound desc = could not find container \"524f212977033a1bd2de88b5b493663bfe5755c1ed253ef4a3df7b5be1378a5b\": container with ID starting with 524f212977033a1bd2de88b5b493663bfe5755c1ed253ef4a3df7b5be1378a5b not found: ID does not exist" Oct 30 18:38:58.070722 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:58.070694 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j"] Oct 30 18:38:58.082205 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:58.082180 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-4dj8j"] Oct 30 18:38:58.129755 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:58.129722 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98"] Oct 30 18:38:58.135407 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:58.135382 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-6xp98"] Oct 30 18:38:58.179517 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:58.179485 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf"] Oct 30 18:38:58.187381 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:58.187356 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-klmcf"] Oct 30 18:38:59.635441 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:59.635407 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3daa8589-9def-486f-82ee-0c923a18b68b" path="/var/lib/kubelet/pods/3daa8589-9def-486f-82ee-0c923a18b68b/volumes" Oct 30 18:38:59.635933 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:59.635918 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c9b7598-cec3-43ab-90cf-93e7b407d68e" path="/var/lib/kubelet/pods/7c9b7598-cec3-43ab-90cf-93e7b407d68e/volumes" Oct 30 18:38:59.636448 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:38:59.636433 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbf4bc50-3670-40de-8159-3d8805fdd601" path="/var/lib/kubelet/pods/bbf4bc50-3670-40de-8159-3d8805fdd601/volumes" Oct 30 18:39:00.385758 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:39:00.385722 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:39:22.729839 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:22.729798 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e77b1d-55ddb87958-l9wgq"] Oct 30 18:39:22.761190 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:22.761159 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e77b1d-9fdccb855-cfxfl"] Oct 30 18:39:27.730521 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:27.730450 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e77b1d-55ddb87958-l9wgq" podUID="c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff" containerName="payload-processing" containerID="cri-o://96d33ad4cbde6ac358f6bc6d784f51c7ea8ad430d5bdbe28109bcc563cef6b34" gracePeriod=25 Oct 30 18:39:27.761767 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:27.761707 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e77b1d-9fdccb855-cfxfl" podUID="67a8b3a3-4d72-4077-875f-a0262982c7e9" containerName="payload-pre-processing" containerID="cri-o://be21ba11cb5649a4342cf5d3d9e4e3776184392c0f31d3e9a1a52036c04a6133" gracePeriod=25 Oct 30 18:39:28.078748 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:28.078720 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e77b1d-55ddb87958-l9wgq" Oct 30 18:39:28.130402 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:28.130368 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e77b1d-9fdccb855-cfxfl" Oct 30 18:39:28.242251 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:28.242216 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff-kube-api-access-2vqtt\" (UniqueName: \"kubernetes.io/projected/c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff-kube-api-access-2vqtt\") pod \"c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff\" (UID: \"c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff\") " Oct 30 18:39:28.242573 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:28.242484 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67a8b3a3-4d72-4077-875f-a0262982c7e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a8b3a3-4d72-4077-875f-a0262982c7e9-plugins-config-volume\") pod \"67a8b3a3-4d72-4077-875f-a0262982c7e9\" (UID: \"67a8b3a3-4d72-4077-875f-a0262982c7e9\") " Oct 30 18:39:28.242573 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:28.242526 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67a8b3a3-4d72-4077-875f-a0262982c7e9-kube-api-access-sppl8\" (UniqueName: \"kubernetes.io/projected/67a8b3a3-4d72-4077-875f-a0262982c7e9-kube-api-access-sppl8\") pod \"67a8b3a3-4d72-4077-875f-a0262982c7e9\" (UID: \"67a8b3a3-4d72-4077-875f-a0262982c7e9\") " Oct 30 18:39:28.242712 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:28.242680 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff-plugins-config-volume\") pod \"c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff\" (UID: \"c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff\") " Oct 30 18:39:28.243221 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:28.243193 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67a8b3a3-4d72-4077-875f-a0262982c7e9-plugins-config-volume" pod "67a8b3a3-4d72-4077-875f-a0262982c7e9" (UID: "67a8b3a3-4d72-4077-875f-a0262982c7e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:39:28.243670 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:28.243632 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff-plugins-config-volume" pod "c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff" (UID: "c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:39:28.244966 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:28.244919 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff-kube-api-access-2vqtt" pod "c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff" (UID: "c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff"). InnerVolumeSpecName "kube-api-access-2vqtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:39:28.245866 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:28.245847 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a8b3a3-4d72-4077-875f-a0262982c7e9-kube-api-access-sppl8" pod "67a8b3a3-4d72-4077-875f-a0262982c7e9" (UID: "67a8b3a3-4d72-4077-875f-a0262982c7e9"). InnerVolumeSpecName "kube-api-access-sppl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:39:28.253877 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:28.253850 4455 generic.go:356] "Generic (PLEG): container finished" podID="c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff" containerID="96d33ad4cbde6ac358f6bc6d784f51c7ea8ad430d5bdbe28109bcc563cef6b34" exitCode=0 Oct 30 18:39:28.254000 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:28.253895 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e77b1d-55ddb87958-l9wgq" Oct 30 18:39:28.254000 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:28.253929 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e77b1d-55ddb87958-l9wgq" event={"ID":"c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff","Type":"ContainerDied","Data":"96d33ad4cbde6ac358f6bc6d784f51c7ea8ad430d5bdbe28109bcc563cef6b34"} Oct 30 18:39:28.254000 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:28.253988 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e77b1d-55ddb87958-l9wgq" event={"ID":"c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff","Type":"ContainerDied","Data":"d20c6dd8b3fa9d0748452cbf744bf6fca73fc423d0c69988c32d37e09db69145"} Oct 30 18:39:28.254176 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:28.254017 4455 scope.go:122] "RemoveContainer" containerID="96d33ad4cbde6ac358f6bc6d784f51c7ea8ad430d5bdbe28109bcc563cef6b34" Oct 30 18:39:28.255723 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:28.255697 4455 generic.go:356] "Generic (PLEG): container finished" podID="67a8b3a3-4d72-4077-875f-a0262982c7e9" containerID="be21ba11cb5649a4342cf5d3d9e4e3776184392c0f31d3e9a1a52036c04a6133" exitCode=0 Oct 30 18:39:28.255834 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:28.255744 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e77b1d-9fdccb855-cfxfl" Oct 30 18:39:28.255834 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:28.255755 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e77b1d-9fdccb855-cfxfl" event={"ID":"67a8b3a3-4d72-4077-875f-a0262982c7e9","Type":"ContainerDied","Data":"be21ba11cb5649a4342cf5d3d9e4e3776184392c0f31d3e9a1a52036c04a6133"} Oct 30 18:39:28.255834 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:28.255824 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e77b1d-9fdccb855-cfxfl" event={"ID":"67a8b3a3-4d72-4077-875f-a0262982c7e9","Type":"ContainerDied","Data":"74cd0b4ecacaf4f55f899279ba26bd1b3171b05243a1f679193e5c81d5f216bb"} Oct 30 18:39:28.265803 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:28.265784 4455 scope.go:122] "RemoveContainer" containerID="96d33ad4cbde6ac358f6bc6d784f51c7ea8ad430d5bdbe28109bcc563cef6b34" Oct 30 18:39:28.266108 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:39:28.266083 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96d33ad4cbde6ac358f6bc6d784f51c7ea8ad430d5bdbe28109bcc563cef6b34\": container with ID starting with 96d33ad4cbde6ac358f6bc6d784f51c7ea8ad430d5bdbe28109bcc563cef6b34 not found: ID does not exist" containerID="96d33ad4cbde6ac358f6bc6d784f51c7ea8ad430d5bdbe28109bcc563cef6b34" Oct 30 18:39:28.266202 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:28.266115 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96d33ad4cbde6ac358f6bc6d784f51c7ea8ad430d5bdbe28109bcc563cef6b34"} err="failed to get container status \"96d33ad4cbde6ac358f6bc6d784f51c7ea8ad430d5bdbe28109bcc563cef6b34\": rpc error: code = NotFound desc = could not find container \"96d33ad4cbde6ac358f6bc6d784f51c7ea8ad430d5bdbe28109bcc563cef6b34\": container with ID starting with 96d33ad4cbde6ac358f6bc6d784f51c7ea8ad430d5bdbe28109bcc563cef6b34 not found: ID does not exist" Oct 30 18:39:28.266202 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:28.266134 4455 scope.go:122] "RemoveContainer" containerID="be21ba11cb5649a4342cf5d3d9e4e3776184392c0f31d3e9a1a52036c04a6133" Oct 30 18:39:28.278181 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:28.278155 4455 scope.go:122] "RemoveContainer" containerID="be21ba11cb5649a4342cf5d3d9e4e3776184392c0f31d3e9a1a52036c04a6133" Oct 30 18:39:28.278445 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:39:28.278426 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be21ba11cb5649a4342cf5d3d9e4e3776184392c0f31d3e9a1a52036c04a6133\": container with ID starting with be21ba11cb5649a4342cf5d3d9e4e3776184392c0f31d3e9a1a52036c04a6133 not found: ID does not exist" containerID="be21ba11cb5649a4342cf5d3d9e4e3776184392c0f31d3e9a1a52036c04a6133" Oct 30 18:39:28.278540 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:28.278451 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be21ba11cb5649a4342cf5d3d9e4e3776184392c0f31d3e9a1a52036c04a6133"} err="failed to get container status \"be21ba11cb5649a4342cf5d3d9e4e3776184392c0f31d3e9a1a52036c04a6133\": rpc error: code = NotFound desc = could not find container \"be21ba11cb5649a4342cf5d3d9e4e3776184392c0f31d3e9a1a52036c04a6133\": container with ID starting with be21ba11cb5649a4342cf5d3d9e4e3776184392c0f31d3e9a1a52036c04a6133 not found: ID does not exist" Oct 30 18:39:28.311210 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:28.311093 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e77b1d-55ddb87958-l9wgq"] Oct 30 18:39:28.315739 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:28.315708 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e77b1d-55ddb87958-l9wgq"] Oct 30 18:39:28.344561 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:28.344522 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:39:28.344561 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:28.344556 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vqtt\" (UniqueName: \"kubernetes.io/projected/c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff-kube-api-access-2vqtt\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:39:28.344783 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:28.344572 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a8b3a3-4d72-4077-875f-a0262982c7e9-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:39:28.344783 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:28.344586 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sppl8\" (UniqueName: \"kubernetes.io/projected/67a8b3a3-4d72-4077-875f-a0262982c7e9-kube-api-access-sppl8\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:39:28.351893 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:28.351854 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e77b1d-9fdccb855-cfxfl"] Oct 30 18:39:28.357497 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:28.357470 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e77b1d-9fdccb855-cfxfl"] Oct 30 18:39:29.634926 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:29.634891 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67a8b3a3-4d72-4077-875f-a0262982c7e9" path="/var/lib/kubelet/pods/67a8b3a3-4d72-4077-875f-a0262982c7e9/volumes" Oct 30 18:39:29.635403 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:29.635389 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff" path="/var/lib/kubelet/pods/c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff/volumes" Oct 30 18:39:30.385858 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:39:30.385816 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:39:37.358618 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:37.358586 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6"] Oct 30 18:39:37.359234 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:37.358797 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6" podUID="fe742589-45cc-4159-b8ba-b4402856f2db" containerName="maas-api" containerID="cri-o://f48aa5cf9f96078a88051f6cf13f60ae1f1a795d1404e4999e5e1acd04e53705" gracePeriod=30 Oct 30 18:39:37.766180 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:37.766152 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6" Oct 30 18:39:37.833394 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:37.831770 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe742589-45cc-4159-b8ba-b4402856f2db-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe742589-45cc-4159-b8ba-b4402856f2db-maas-api-tls\") pod \"fe742589-45cc-4159-b8ba-b4402856f2db\" (UID: \"fe742589-45cc-4159-b8ba-b4402856f2db\") " Oct 30 18:39:37.833394 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:37.832032 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe742589-45cc-4159-b8ba-b4402856f2db-kube-api-access-2znqm\" (UniqueName: \"kubernetes.io/projected/fe742589-45cc-4159-b8ba-b4402856f2db-kube-api-access-2znqm\") pod \"fe742589-45cc-4159-b8ba-b4402856f2db\" (UID: \"fe742589-45cc-4159-b8ba-b4402856f2db\") " Oct 30 18:39:37.833394 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:37.832094 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe742589-45cc-4159-b8ba-b4402856f2db-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe742589-45cc-4159-b8ba-b4402856f2db-metrics-certs\") pod \"fe742589-45cc-4159-b8ba-b4402856f2db\" (UID: \"fe742589-45cc-4159-b8ba-b4402856f2db\") " Oct 30 18:39:37.833394 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:37.832146 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe742589-45cc-4159-b8ba-b4402856f2db-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe742589-45cc-4159-b8ba-b4402856f2db-openshift-service-ca\") pod \"fe742589-45cc-4159-b8ba-b4402856f2db\" (UID: \"fe742589-45cc-4159-b8ba-b4402856f2db\") " Oct 30 18:39:37.853561 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:37.853511 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe742589-45cc-4159-b8ba-b4402856f2db-metrics-certs" pod "fe742589-45cc-4159-b8ba-b4402856f2db" (UID: "fe742589-45cc-4159-b8ba-b4402856f2db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:39:37.853763 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:37.853746 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe742589-45cc-4159-b8ba-b4402856f2db-openshift-service-ca" pod "fe742589-45cc-4159-b8ba-b4402856f2db" (UID: "fe742589-45cc-4159-b8ba-b4402856f2db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:39:37.865692 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:37.865649 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe742589-45cc-4159-b8ba-b4402856f2db-kube-api-access-2znqm" pod "fe742589-45cc-4159-b8ba-b4402856f2db" (UID: "fe742589-45cc-4159-b8ba-b4402856f2db"). InnerVolumeSpecName "kube-api-access-2znqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:39:37.873867 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:37.873811 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe742589-45cc-4159-b8ba-b4402856f2db-maas-api-tls" pod "fe742589-45cc-4159-b8ba-b4402856f2db" (UID: "fe742589-45cc-4159-b8ba-b4402856f2db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:39:37.934290 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:37.934105 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2znqm\" (UniqueName: \"kubernetes.io/projected/fe742589-45cc-4159-b8ba-b4402856f2db-kube-api-access-2znqm\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:39:37.934290 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:37.934145 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe742589-45cc-4159-b8ba-b4402856f2db-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:39:37.934290 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:37.934160 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe742589-45cc-4159-b8ba-b4402856f2db-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:39:37.934290 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:37.934172 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe742589-45cc-4159-b8ba-b4402856f2db-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:39:37.957633 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:37.957551 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg"] Oct 30 18:39:37.957925 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:37.957837 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg" podUID="4deaf1e2-755d-41bb-b3b7-ff08a88186d0" containerName="maas-api" containerID="cri-o://9e8dbac784cd68944e95fa0823f47667690ca446f857d451557058810923604c" gracePeriod=30 Oct 30 18:39:38.039316 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:38.038710 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-50e9fc-647b8c7b4-78rvk"] Oct 30 18:39:38.061256 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:38.061226 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-50e9fc-56cdb5789f-h86nr"] Oct 30 18:39:38.090363 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:39:38.090169 4455 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4deaf1e2_755d_41bb_b3b7_ff08a88186d0.slice/crio-conmon-9e8dbac784cd68944e95fa0823f47667690ca446f857d451557058810923604c.scope\": RecentStats: unable to find data in memory cache]" Oct 30 18:39:38.266891 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:38.266863 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg" Oct 30 18:39:38.342306 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:38.342203 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4deaf1e2-755d-41bb-b3b7-ff08a88186d0-kube-api-access-b4v5x\" (UniqueName: \"kubernetes.io/projected/4deaf1e2-755d-41bb-b3b7-ff08a88186d0-kube-api-access-b4v5x\") pod \"4deaf1e2-755d-41bb-b3b7-ff08a88186d0\" (UID: \"4deaf1e2-755d-41bb-b3b7-ff08a88186d0\") " Oct 30 18:39:38.342510 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:38.342392 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4deaf1e2-755d-41bb-b3b7-ff08a88186d0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4deaf1e2-755d-41bb-b3b7-ff08a88186d0-openshift-service-ca\") pod \"4deaf1e2-755d-41bb-b3b7-ff08a88186d0\" (UID: \"4deaf1e2-755d-41bb-b3b7-ff08a88186d0\") " Oct 30 18:39:38.342605 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:38.342539 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4deaf1e2-755d-41bb-b3b7-ff08a88186d0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4deaf1e2-755d-41bb-b3b7-ff08a88186d0-maas-api-tls\") pod \"4deaf1e2-755d-41bb-b3b7-ff08a88186d0\" (UID: \"4deaf1e2-755d-41bb-b3b7-ff08a88186d0\") " Oct 30 18:39:38.342673 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:38.342605 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4deaf1e2-755d-41bb-b3b7-ff08a88186d0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4deaf1e2-755d-41bb-b3b7-ff08a88186d0-metrics-certs\") pod \"4deaf1e2-755d-41bb-b3b7-ff08a88186d0\" (UID: \"4deaf1e2-755d-41bb-b3b7-ff08a88186d0\") " Oct 30 18:39:38.342875 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:38.342843 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4deaf1e2-755d-41bb-b3b7-ff08a88186d0-openshift-service-ca" pod "4deaf1e2-755d-41bb-b3b7-ff08a88186d0" (UID: "4deaf1e2-755d-41bb-b3b7-ff08a88186d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:39:38.343265 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:38.343245 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4deaf1e2-755d-41bb-b3b7-ff08a88186d0-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:39:38.345568 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:38.345533 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4deaf1e2-755d-41bb-b3b7-ff08a88186d0-kube-api-access-b4v5x" pod "4deaf1e2-755d-41bb-b3b7-ff08a88186d0" (UID: "4deaf1e2-755d-41bb-b3b7-ff08a88186d0"). InnerVolumeSpecName "kube-api-access-b4v5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:39:38.345691 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:38.345585 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4deaf1e2-755d-41bb-b3b7-ff08a88186d0-maas-api-tls" pod "4deaf1e2-755d-41bb-b3b7-ff08a88186d0" (UID: "4deaf1e2-755d-41bb-b3b7-ff08a88186d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:39:38.345691 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:38.345670 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4deaf1e2-755d-41bb-b3b7-ff08a88186d0-metrics-certs" pod "4deaf1e2-755d-41bb-b3b7-ff08a88186d0" (UID: "4deaf1e2-755d-41bb-b3b7-ff08a88186d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:39:38.423469 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:38.423374 4455 generic.go:356] "Generic (PLEG): container finished" podID="4deaf1e2-755d-41bb-b3b7-ff08a88186d0" containerID="9e8dbac784cd68944e95fa0823f47667690ca446f857d451557058810923604c" exitCode=0 Oct 30 18:39:38.423469 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:38.423468 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg" Oct 30 18:39:38.424022 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:38.423514 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg" event={"ID":"4deaf1e2-755d-41bb-b3b7-ff08a88186d0","Type":"ContainerDied","Data":"9e8dbac784cd68944e95fa0823f47667690ca446f857d451557058810923604c"} Oct 30 18:39:38.424022 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:38.423589 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg" event={"ID":"4deaf1e2-755d-41bb-b3b7-ff08a88186d0","Type":"ContainerDied","Data":"1f454d2cfeb70acf4b0049ca3187593866cb3882a8ea18c4332b9696e0e028af"} Oct 30 18:39:38.424022 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:38.423628 4455 scope.go:122] "RemoveContainer" containerID="9e8dbac784cd68944e95fa0823f47667690ca446f857d451557058810923604c" Oct 30 18:39:38.433939 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:38.431104 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6" Oct 30 18:39:38.451637 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:38.443166 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6" event={"ID":"fe742589-45cc-4159-b8ba-b4402856f2db","Type":"ContainerDied","Data":"f48aa5cf9f96078a88051f6cf13f60ae1f1a795d1404e4999e5e1acd04e53705"} Oct 30 18:39:38.451637 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:38.450511 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4v5x\" (UniqueName: \"kubernetes.io/projected/4deaf1e2-755d-41bb-b3b7-ff08a88186d0-kube-api-access-b4v5x\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:39:38.451637 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:38.450544 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4deaf1e2-755d-41bb-b3b7-ff08a88186d0-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:39:38.451637 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:38.450557 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4deaf1e2-755d-41bb-b3b7-ff08a88186d0-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:39:38.513715 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:38.469559 4455 scope.go:122] "RemoveContainer" containerID="9e8dbac784cd68944e95fa0823f47667690ca446f857d451557058810923604c" Oct 30 18:39:38.513715 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:39:38.481076 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e8dbac784cd68944e95fa0823f47667690ca446f857d451557058810923604c\": container with ID starting with 9e8dbac784cd68944e95fa0823f47667690ca446f857d451557058810923604c not found: ID does not exist" containerID="9e8dbac784cd68944e95fa0823f47667690ca446f857d451557058810923604c" Oct 30 18:39:38.513715 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:38.481121 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e8dbac784cd68944e95fa0823f47667690ca446f857d451557058810923604c"} err="failed to get container status \"9e8dbac784cd68944e95fa0823f47667690ca446f857d451557058810923604c\": rpc error: code = NotFound desc = could not find container \"9e8dbac784cd68944e95fa0823f47667690ca446f857d451557058810923604c\": container with ID starting with 9e8dbac784cd68944e95fa0823f47667690ca446f857d451557058810923604c not found: ID does not exist" Oct 30 18:39:38.513715 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:38.481152 4455 scope.go:122] "RemoveContainer" containerID="f48aa5cf9f96078a88051f6cf13f60ae1f1a795d1404e4999e5e1acd04e53705" Oct 30 18:39:38.513715 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:38.488491 4455 generic.go:356] "Generic (PLEG): container finished" podID="fe742589-45cc-4159-b8ba-b4402856f2db" containerID="f48aa5cf9f96078a88051f6cf13f60ae1f1a795d1404e4999e5e1acd04e53705" exitCode=0 Oct 30 18:39:38.513715 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:38.488558 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6" event={"ID":"fe742589-45cc-4159-b8ba-b4402856f2db","Type":"ContainerDied","Data":"4f78a05c3de3a84bc14563c87b8dbf23917483251bd7afa6a7befd3a28f3d15c"} Oct 30 18:39:38.571454 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:38.571414 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg"] Oct 30 18:39:38.572510 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:38.572480 4455 scope.go:122] "RemoveContainer" containerID="f48aa5cf9f96078a88051f6cf13f60ae1f1a795d1404e4999e5e1acd04e53705" Oct 30 18:39:38.575951 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:38.575920 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-50e9fc-6dc4d6fbff-p2bfg"] Oct 30 18:39:38.576726 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:39:38.576702 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f48aa5cf9f96078a88051f6cf13f60ae1f1a795d1404e4999e5e1acd04e53705\": container with ID starting with f48aa5cf9f96078a88051f6cf13f60ae1f1a795d1404e4999e5e1acd04e53705 not found: ID does not exist" containerID="f48aa5cf9f96078a88051f6cf13f60ae1f1a795d1404e4999e5e1acd04e53705" Oct 30 18:39:38.576820 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:38.576732 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f48aa5cf9f96078a88051f6cf13f60ae1f1a795d1404e4999e5e1acd04e53705"} err="failed to get container status \"f48aa5cf9f96078a88051f6cf13f60ae1f1a795d1404e4999e5e1acd04e53705\": rpc error: code = NotFound desc = could not find container \"f48aa5cf9f96078a88051f6cf13f60ae1f1a795d1404e4999e5e1acd04e53705\": container with ID starting with f48aa5cf9f96078a88051f6cf13f60ae1f1a795d1404e4999e5e1acd04e53705 not found: ID does not exist" Oct 30 18:39:38.595222 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:38.595133 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6"] Oct 30 18:39:38.599667 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:38.599621 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e77b1d-65d6845464-257s6"] Oct 30 18:39:39.634614 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:39.634583 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4deaf1e2-755d-41bb-b3b7-ff08a88186d0" path="/var/lib/kubelet/pods/4deaf1e2-755d-41bb-b3b7-ff08a88186d0/volumes" Oct 30 18:39:39.635356 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:39.635052 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe742589-45cc-4159-b8ba-b4402856f2db" path="/var/lib/kubelet/pods/fe742589-45cc-4159-b8ba-b4402856f2db/volumes" Oct 30 18:39:43.039810 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:43.039748 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-50e9fc-647b8c7b4-78rvk" podUID="67ec911d-b41b-49f5-b1e3-4637bc7c48f3" containerName="payload-processing" containerID="cri-o://ce2473c9edf9cd1a6177ceaee91cd4eb7a0a8e83daf7d2a046a45957ee245a77" gracePeriod=25 Oct 30 18:39:43.062114 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:43.062077 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-50e9fc-56cdb5789f-h86nr" podUID="13f59414-113d-4863-88b8-e43c25b17cff" containerName="payload-pre-processing" containerID="cri-o://b8f720cdbb798b5e7eaa1831d6d61d8af1e1bbd00066cc9b982cbfa3f6b2b049" gracePeriod=25 Oct 30 18:39:43.365585 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:43.365553 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-50e9fc-647b8c7b4-78rvk" Oct 30 18:39:43.403756 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:43.403728 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-50e9fc-56cdb5789f-h86nr" Oct 30 18:39:43.446220 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:43.446190 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67ec911d-b41b-49f5-b1e3-4637bc7c48f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ec911d-b41b-49f5-b1e3-4637bc7c48f3-plugins-config-volume\") pod \"67ec911d-b41b-49f5-b1e3-4637bc7c48f3\" (UID: \"67ec911d-b41b-49f5-b1e3-4637bc7c48f3\") " Oct 30 18:39:43.446220 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:43.446227 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67ec911d-b41b-49f5-b1e3-4637bc7c48f3-kube-api-access-kz28f\" (UniqueName: \"kubernetes.io/projected/67ec911d-b41b-49f5-b1e3-4637bc7c48f3-kube-api-access-kz28f\") pod \"67ec911d-b41b-49f5-b1e3-4637bc7c48f3\" (UID: \"67ec911d-b41b-49f5-b1e3-4637bc7c48f3\") " Oct 30 18:39:43.446624 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:43.446597 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67ec911d-b41b-49f5-b1e3-4637bc7c48f3-plugins-config-volume" pod "67ec911d-b41b-49f5-b1e3-4637bc7c48f3" (UID: "67ec911d-b41b-49f5-b1e3-4637bc7c48f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:39:43.446744 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:43.446729 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ec911d-b41b-49f5-b1e3-4637bc7c48f3-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:39:43.448904 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:43.448878 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ec911d-b41b-49f5-b1e3-4637bc7c48f3-kube-api-access-kz28f" pod "67ec911d-b41b-49f5-b1e3-4637bc7c48f3" (UID: "67ec911d-b41b-49f5-b1e3-4637bc7c48f3"). InnerVolumeSpecName "kube-api-access-kz28f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:39:43.547592 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:43.547564 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13f59414-113d-4863-88b8-e43c25b17cff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13f59414-113d-4863-88b8-e43c25b17cff-plugins-config-volume\") pod \"13f59414-113d-4863-88b8-e43c25b17cff\" (UID: \"13f59414-113d-4863-88b8-e43c25b17cff\") " Oct 30 18:39:43.547802 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:43.547710 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13f59414-113d-4863-88b8-e43c25b17cff-kube-api-access-srcm8\" (UniqueName: \"kubernetes.io/projected/13f59414-113d-4863-88b8-e43c25b17cff-kube-api-access-srcm8\") pod \"13f59414-113d-4863-88b8-e43c25b17cff\" (UID: \"13f59414-113d-4863-88b8-e43c25b17cff\") " Oct 30 18:39:43.548062 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:43.548019 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13f59414-113d-4863-88b8-e43c25b17cff-plugins-config-volume" pod "13f59414-113d-4863-88b8-e43c25b17cff" (UID: "13f59414-113d-4863-88b8-e43c25b17cff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:39:43.548226 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:43.548209 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13f59414-113d-4863-88b8-e43c25b17cff-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:39:43.548288 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:43.548229 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz28f\" (UniqueName: \"kubernetes.io/projected/67ec911d-b41b-49f5-b1e3-4637bc7c48f3-kube-api-access-kz28f\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:39:43.549471 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:43.549444 4455 generic.go:356] "Generic (PLEG): container finished" podID="13f59414-113d-4863-88b8-e43c25b17cff" containerID="b8f720cdbb798b5e7eaa1831d6d61d8af1e1bbd00066cc9b982cbfa3f6b2b049" exitCode=0 Oct 30 18:39:43.549595 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:43.549494 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-50e9fc-56cdb5789f-h86nr" Oct 30 18:39:43.549595 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:43.549538 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-50e9fc-56cdb5789f-h86nr" event={"ID":"13f59414-113d-4863-88b8-e43c25b17cff","Type":"ContainerDied","Data":"b8f720cdbb798b5e7eaa1831d6d61d8af1e1bbd00066cc9b982cbfa3f6b2b049"} Oct 30 18:39:43.549739 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:43.549599 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-50e9fc-56cdb5789f-h86nr" event={"ID":"13f59414-113d-4863-88b8-e43c25b17cff","Type":"ContainerDied","Data":"072b6d4bfbc99fe99eb886328ffe2ec2d5c1b23ac89e5c6d6391f782ac19468c"} Oct 30 18:39:43.549739 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:43.549632 4455 scope.go:122] "RemoveContainer" containerID="b8f720cdbb798b5e7eaa1831d6d61d8af1e1bbd00066cc9b982cbfa3f6b2b049" Oct 30 18:39:43.550299 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:43.550276 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f59414-113d-4863-88b8-e43c25b17cff-kube-api-access-srcm8" pod "13f59414-113d-4863-88b8-e43c25b17cff" (UID: "13f59414-113d-4863-88b8-e43c25b17cff"). InnerVolumeSpecName "kube-api-access-srcm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:39:43.551026 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:43.551007 4455 generic.go:356] "Generic (PLEG): container finished" podID="67ec911d-b41b-49f5-b1e3-4637bc7c48f3" containerID="ce2473c9edf9cd1a6177ceaee91cd4eb7a0a8e83daf7d2a046a45957ee245a77" exitCode=0 Oct 30 18:39:43.551113 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:43.551049 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-50e9fc-647b8c7b4-78rvk" Oct 30 18:39:43.551162 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:43.551110 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-50e9fc-647b8c7b4-78rvk" event={"ID":"67ec911d-b41b-49f5-b1e3-4637bc7c48f3","Type":"ContainerDied","Data":"ce2473c9edf9cd1a6177ceaee91cd4eb7a0a8e83daf7d2a046a45957ee245a77"} Oct 30 18:39:43.551162 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:43.551154 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-50e9fc-647b8c7b4-78rvk" event={"ID":"67ec911d-b41b-49f5-b1e3-4637bc7c48f3","Type":"ContainerDied","Data":"c8f686f8893110d637ab2e2e74db0cc58afb4c4f6edc49930a050115a51efbf4"} Oct 30 18:39:43.560511 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:43.560490 4455 scope.go:122] "RemoveContainer" containerID="b8f720cdbb798b5e7eaa1831d6d61d8af1e1bbd00066cc9b982cbfa3f6b2b049" Oct 30 18:39:43.560846 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:39:43.560827 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8f720cdbb798b5e7eaa1831d6d61d8af1e1bbd00066cc9b982cbfa3f6b2b049\": container with ID starting with b8f720cdbb798b5e7eaa1831d6d61d8af1e1bbd00066cc9b982cbfa3f6b2b049 not found: ID does not exist" containerID="b8f720cdbb798b5e7eaa1831d6d61d8af1e1bbd00066cc9b982cbfa3f6b2b049" Oct 30 18:39:43.560898 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:43.560857 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8f720cdbb798b5e7eaa1831d6d61d8af1e1bbd00066cc9b982cbfa3f6b2b049"} err="failed to get container status \"b8f720cdbb798b5e7eaa1831d6d61d8af1e1bbd00066cc9b982cbfa3f6b2b049\": rpc error: code = NotFound desc = could not find container \"b8f720cdbb798b5e7eaa1831d6d61d8af1e1bbd00066cc9b982cbfa3f6b2b049\": container with ID starting with b8f720cdbb798b5e7eaa1831d6d61d8af1e1bbd00066cc9b982cbfa3f6b2b049 not found: ID does not exist" Oct 30 18:39:43.560898 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:43.560879 4455 scope.go:122] "RemoveContainer" containerID="ce2473c9edf9cd1a6177ceaee91cd4eb7a0a8e83daf7d2a046a45957ee245a77" Oct 30 18:39:43.574033 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:43.574005 4455 scope.go:122] "RemoveContainer" containerID="ce2473c9edf9cd1a6177ceaee91cd4eb7a0a8e83daf7d2a046a45957ee245a77" Oct 30 18:39:43.574514 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:39:43.574479 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce2473c9edf9cd1a6177ceaee91cd4eb7a0a8e83daf7d2a046a45957ee245a77\": container with ID starting with ce2473c9edf9cd1a6177ceaee91cd4eb7a0a8e83daf7d2a046a45957ee245a77 not found: ID does not exist" containerID="ce2473c9edf9cd1a6177ceaee91cd4eb7a0a8e83daf7d2a046a45957ee245a77" Oct 30 18:39:43.574662 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:43.574524 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce2473c9edf9cd1a6177ceaee91cd4eb7a0a8e83daf7d2a046a45957ee245a77"} err="failed to get container status \"ce2473c9edf9cd1a6177ceaee91cd4eb7a0a8e83daf7d2a046a45957ee245a77\": rpc error: code = NotFound desc = could not find container \"ce2473c9edf9cd1a6177ceaee91cd4eb7a0a8e83daf7d2a046a45957ee245a77\": container with ID starting with ce2473c9edf9cd1a6177ceaee91cd4eb7a0a8e83daf7d2a046a45957ee245a77 not found: ID does not exist" Oct 30 18:39:43.608299 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:43.608206 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-50e9fc-647b8c7b4-78rvk"] Oct 30 18:39:43.615246 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:43.615202 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-50e9fc-647b8c7b4-78rvk"] Oct 30 18:39:43.635290 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:43.635255 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67ec911d-b41b-49f5-b1e3-4637bc7c48f3" path="/var/lib/kubelet/pods/67ec911d-b41b-49f5-b1e3-4637bc7c48f3/volumes" Oct 30 18:39:43.649997 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:43.649968 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srcm8\" (UniqueName: \"kubernetes.io/projected/13f59414-113d-4863-88b8-e43c25b17cff-kube-api-access-srcm8\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:39:43.897136 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:43.896216 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-50e9fc-56cdb5789f-h86nr"] Oct 30 18:39:43.901959 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:43.901919 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-50e9fc-56cdb5789f-h86nr"] Oct 30 18:39:45.635174 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:45.635136 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13f59414-113d-4863-88b8-e43c25b17cff" path="/var/lib/kubelet/pods/13f59414-113d-4863-88b8-e43c25b17cff/volumes" Oct 30 18:39:46.303309 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:46.303263 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7"] Oct 30 18:39:46.303577 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:46.303543 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" podUID="099a77ed-9f6a-4937-85c1-b7053bc602fa" containerName="istio-proxy" containerID="cri-o://7ff97b59aafc88f59908b1532470bfaf7408787498b324608e2a5fb88b8d7d43" gracePeriod=30 Oct 30 18:39:51.589892 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.589825 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" Oct 30 18:39:51.624245 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.624213 4455 generic.go:356] "Generic (PLEG): container finished" podID="099a77ed-9f6a-4937-85c1-b7053bc602fa" containerID="7ff97b59aafc88f59908b1532470bfaf7408787498b324608e2a5fb88b8d7d43" exitCode=0 Oct 30 18:39:51.624459 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.624290 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" Oct 30 18:39:51.624545 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.624486 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" event={"ID":"099a77ed-9f6a-4937-85c1-b7053bc602fa","Type":"ContainerDied","Data":"7ff97b59aafc88f59908b1532470bfaf7408787498b324608e2a5fb88b8d7d43"} Oct 30 18:39:51.624610 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.624547 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7" event={"ID":"099a77ed-9f6a-4937-85c1-b7053bc602fa","Type":"ContainerDied","Data":"083b64e5ddb789b80dccd98f1a02f460e2e8dcb1e72852f3ade88727c7a92a67"} Oct 30 18:39:51.624610 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.624581 4455 scope.go:122] "RemoveContainer" containerID="7ff97b59aafc88f59908b1532470bfaf7408787498b324608e2a5fb88b8d7d43" Oct 30 18:39:51.637561 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.637529 4455 scope.go:122] "RemoveContainer" containerID="7ff97b59aafc88f59908b1532470bfaf7408787498b324608e2a5fb88b8d7d43" Oct 30 18:39:51.637842 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:39:51.637819 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ff97b59aafc88f59908b1532470bfaf7408787498b324608e2a5fb88b8d7d43\": container with ID starting with 7ff97b59aafc88f59908b1532470bfaf7408787498b324608e2a5fb88b8d7d43 not found: ID does not exist" containerID="7ff97b59aafc88f59908b1532470bfaf7408787498b324608e2a5fb88b8d7d43" Oct 30 18:39:51.637935 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.637853 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ff97b59aafc88f59908b1532470bfaf7408787498b324608e2a5fb88b8d7d43"} err="failed to get container status \"7ff97b59aafc88f59908b1532470bfaf7408787498b324608e2a5fb88b8d7d43\": rpc error: code = NotFound desc = could not find container \"7ff97b59aafc88f59908b1532470bfaf7408787498b324608e2a5fb88b8d7d43\": container with ID starting with 7ff97b59aafc88f59908b1532470bfaf7408787498b324608e2a5fb88b8d7d43 not found: ID does not exist" Oct 30 18:39:51.694135 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.694102 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/099a77ed-9f6a-4937-85c1-b7053bc602fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099a77ed-9f6a-4937-85c1-b7053bc602fa-istio-data\") pod \"099a77ed-9f6a-4937-85c1-b7053bc602fa\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " Oct 30 18:39:51.694401 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.694172 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/099a77ed-9f6a-4937-85c1-b7053bc602fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099a77ed-9f6a-4937-85c1-b7053bc602fa-istio-envoy\") pod \"099a77ed-9f6a-4937-85c1-b7053bc602fa\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " Oct 30 18:39:51.694401 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.694226 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/099a77ed-9f6a-4937-85c1-b7053bc602fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099a77ed-9f6a-4937-85c1-b7053bc602fa-credential-socket\") pod \"099a77ed-9f6a-4937-85c1-b7053bc602fa\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " Oct 30 18:39:51.694401 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.694291 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/099a77ed-9f6a-4937-85c1-b7053bc602fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099a77ed-9f6a-4937-85c1-b7053bc602fa-istio-podinfo\") pod \"099a77ed-9f6a-4937-85c1-b7053bc602fa\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " Oct 30 18:39:51.694602 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.694482 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/099a77ed-9f6a-4937-85c1-b7053bc602fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099a77ed-9f6a-4937-85c1-b7053bc602fa-workload-socket\") pod \"099a77ed-9f6a-4937-85c1-b7053bc602fa\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " Oct 30 18:39:51.694602 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.694525 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/099a77ed-9f6a-4937-85c1-b7053bc602fa-istio-token\" (UniqueName: \"kubernetes.io/projected/099a77ed-9f6a-4937-85c1-b7053bc602fa-istio-token\") pod \"099a77ed-9f6a-4937-85c1-b7053bc602fa\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " Oct 30 18:39:51.694712 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.694618 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/099a77ed-9f6a-4937-85c1-b7053bc602fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099a77ed-9f6a-4937-85c1-b7053bc602fa-workload-certs\") pod \"099a77ed-9f6a-4937-85c1-b7053bc602fa\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " Oct 30 18:39:51.694712 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.694650 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/099a77ed-9f6a-4937-85c1-b7053bc602fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099a77ed-9f6a-4937-85c1-b7053bc602fa-istiod-ca-cert\") pod \"099a77ed-9f6a-4937-85c1-b7053bc602fa\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " Oct 30 18:39:51.694812 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.694731 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/099a77ed-9f6a-4937-85c1-b7053bc602fa-kube-api-access-fq25n\" (UniqueName: \"kubernetes.io/projected/099a77ed-9f6a-4937-85c1-b7053bc602fa-kube-api-access-fq25n\") pod \"099a77ed-9f6a-4937-85c1-b7053bc602fa\" (UID: \"099a77ed-9f6a-4937-85c1-b7053bc602fa\") " Oct 30 18:39:51.695587 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.695557 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099a77ed-9f6a-4937-85c1-b7053bc602fa-workload-certs" pod "099a77ed-9f6a-4937-85c1-b7053bc602fa" (UID: "099a77ed-9f6a-4937-85c1-b7053bc602fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:51.695745 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.695574 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099a77ed-9f6a-4937-85c1-b7053bc602fa-istio-data" pod "099a77ed-9f6a-4937-85c1-b7053bc602fa" (UID: "099a77ed-9f6a-4937-85c1-b7053bc602fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:51.696186 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.696155 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099a77ed-9f6a-4937-85c1-b7053bc602fa-credential-socket" pod "099a77ed-9f6a-4937-85c1-b7053bc602fa" (UID: "099a77ed-9f6a-4937-85c1-b7053bc602fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:51.696297 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.696223 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099a77ed-9f6a-4937-85c1-b7053bc602fa-workload-socket" pod "099a77ed-9f6a-4937-85c1-b7053bc602fa" (UID: "099a77ed-9f6a-4937-85c1-b7053bc602fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:51.696670 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.696646 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/099a77ed-9f6a-4937-85c1-b7053bc602fa-istiod-ca-cert" pod "099a77ed-9f6a-4937-85c1-b7053bc602fa" (UID: "099a77ed-9f6a-4937-85c1-b7053bc602fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:39:51.697185 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.697151 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099a77ed-9f6a-4937-85c1-b7053bc602fa-istio-envoy" pod "099a77ed-9f6a-4937-85c1-b7053bc602fa" (UID: "099a77ed-9f6a-4937-85c1-b7053bc602fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:51.698364 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.698305 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/099a77ed-9f6a-4937-85c1-b7053bc602fa-istio-podinfo" pod "099a77ed-9f6a-4937-85c1-b7053bc602fa" (UID: "099a77ed-9f6a-4937-85c1-b7053bc602fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:39:51.698459 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.698380 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/099a77ed-9f6a-4937-85c1-b7053bc602fa-kube-api-access-fq25n" pod "099a77ed-9f6a-4937-85c1-b7053bc602fa" (UID: "099a77ed-9f6a-4937-85c1-b7053bc602fa"). InnerVolumeSpecName "kube-api-access-fq25n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:39:51.703282 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.703252 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/099a77ed-9f6a-4937-85c1-b7053bc602fa-istio-token" pod "099a77ed-9f6a-4937-85c1-b7053bc602fa" (UID: "099a77ed-9f6a-4937-85c1-b7053bc602fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:39:51.796843 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.796802 4455 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099a77ed-9f6a-4937-85c1-b7053bc602fa-workload-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:39:51.796843 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.796834 4455 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099a77ed-9f6a-4937-85c1-b7053bc602fa-istio-token\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:39:51.796843 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.796849 4455 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099a77ed-9f6a-4937-85c1-b7053bc602fa-workload-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:39:51.797097 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.796860 4455 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099a77ed-9f6a-4937-85c1-b7053bc602fa-istiod-ca-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:39:51.797097 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.796871 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq25n\" (UniqueName: \"kubernetes.io/projected/099a77ed-9f6a-4937-85c1-b7053bc602fa-kube-api-access-fq25n\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:39:51.797097 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.796884 4455 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099a77ed-9f6a-4937-85c1-b7053bc602fa-istio-data\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:39:51.797097 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.796895 4455 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099a77ed-9f6a-4937-85c1-b7053bc602fa-istio-envoy\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:39:51.797097 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.796905 4455 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099a77ed-9f6a-4937-85c1-b7053bc602fa-credential-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:39:51.797097 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.796915 4455 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099a77ed-9f6a-4937-85c1-b7053bc602fa-istio-podinfo\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:39:51.980985 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.980951 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7"] Oct 30 18:39:51.988881 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:51.988850 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-4c8b7"] Oct 30 18:39:53.634999 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:39:53.634960 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="099a77ed-9f6a-4937-85c1-b7053bc602fa" path="/var/lib/kubelet/pods/099a77ed-9f6a-4937-85c1-b7053bc602fa/volumes" Oct 30 18:40:00.423676 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:40:00.423456 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:40:01.814268 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:01.814193 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n"] Oct 30 18:40:01.815336 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:01.815285 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n" podUID="a11771b6-cb92-4783-bd70-4eb964d94f00" containerName="maas-api" containerID="cri-o://da55ebe22dd5589f70141c2b75d52cf3090a6b3b4a969324a1a3067c471199c9" gracePeriod=30 Oct 30 18:40:02.972676 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:02.961338 4455 generic.go:356] "Generic (PLEG): container finished" podID="a11771b6-cb92-4783-bd70-4eb964d94f00" containerID="da55ebe22dd5589f70141c2b75d52cf3090a6b3b4a969324a1a3067c471199c9" exitCode=0 Oct 30 18:40:02.972676 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:02.961560 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n" event={"ID":"a11771b6-cb92-4783-bd70-4eb964d94f00","Type":"ContainerDied","Data":"da55ebe22dd5589f70141c2b75d52cf3090a6b3b4a969324a1a3067c471199c9"} Oct 30 18:40:03.456835 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:03.456480 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n" Oct 30 18:40:03.457146 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:03.457115 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a11771b6-cb92-4783-bd70-4eb964d94f00-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a11771b6-cb92-4783-bd70-4eb964d94f00-maas-api-tls\") pod \"a11771b6-cb92-4783-bd70-4eb964d94f00\" (UID: \"a11771b6-cb92-4783-bd70-4eb964d94f00\") " Oct 30 18:40:03.457420 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:03.457398 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a11771b6-cb92-4783-bd70-4eb964d94f00-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a11771b6-cb92-4783-bd70-4eb964d94f00-openshift-service-ca\") pod \"a11771b6-cb92-4783-bd70-4eb964d94f00\" (UID: \"a11771b6-cb92-4783-bd70-4eb964d94f00\") " Oct 30 18:40:03.458526 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:03.458491 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a11771b6-cb92-4783-bd70-4eb964d94f00-openshift-service-ca" pod "a11771b6-cb92-4783-bd70-4eb964d94f00" (UID: "a11771b6-cb92-4783-bd70-4eb964d94f00"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:03.582001 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:03.581916 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a11771b6-cb92-4783-bd70-4eb964d94f00-maas-api-tls" pod "a11771b6-cb92-4783-bd70-4eb964d94f00" (UID: "a11771b6-cb92-4783-bd70-4eb964d94f00"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:03.582428 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:03.582408 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a11771b6-cb92-4783-bd70-4eb964d94f00-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a11771b6-cb92-4783-bd70-4eb964d94f00-metrics-certs\") pod \"a11771b6-cb92-4783-bd70-4eb964d94f00\" (UID: \"a11771b6-cb92-4783-bd70-4eb964d94f00\") " Oct 30 18:40:03.582952 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:03.582933 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a11771b6-cb92-4783-bd70-4eb964d94f00-kube-api-access-p77tf\" (UniqueName: \"kubernetes.io/projected/a11771b6-cb92-4783-bd70-4eb964d94f00-kube-api-access-p77tf\") pod \"a11771b6-cb92-4783-bd70-4eb964d94f00\" (UID: \"a11771b6-cb92-4783-bd70-4eb964d94f00\") " Oct 30 18:40:03.583657 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:03.583638 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a11771b6-cb92-4783-bd70-4eb964d94f00-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a11771b6-cb92-4783-bd70-4eb964d94f00-maas-api-tls\") pod \"a11771b6-cb92-4783-bd70-4eb964d94f00\" (UID: \"a11771b6-cb92-4783-bd70-4eb964d94f00\") " Oct 30 18:40:03.583957 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:40:03.583839 4455 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/a11771b6-cb92-4783-bd70-4eb964d94f00/volumes/kubernetes.io~secret/maas-api-tls Oct 30 18:40:03.583957 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:03.583933 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a11771b6-cb92-4783-bd70-4eb964d94f00-maas-api-tls" pod "a11771b6-cb92-4783-bd70-4eb964d94f00" (UID: "a11771b6-cb92-4783-bd70-4eb964d94f00"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:03.584632 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:03.584613 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a11771b6-cb92-4783-bd70-4eb964d94f00-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:03.584753 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:03.584742 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a11771b6-cb92-4783-bd70-4eb964d94f00-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:03.597010 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:03.596940 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a11771b6-cb92-4783-bd70-4eb964d94f00-kube-api-access-p77tf" pod "a11771b6-cb92-4783-bd70-4eb964d94f00" (UID: "a11771b6-cb92-4783-bd70-4eb964d94f00"). InnerVolumeSpecName "kube-api-access-p77tf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:03.603402 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:03.603361 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a11771b6-cb92-4783-bd70-4eb964d94f00-metrics-certs" pod "a11771b6-cb92-4783-bd70-4eb964d94f00" (UID: "a11771b6-cb92-4783-bd70-4eb964d94f00"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:03.736369 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:03.693942 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a11771b6-cb92-4783-bd70-4eb964d94f00-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:03.736369 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:03.693979 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p77tf\" (UniqueName: \"kubernetes.io/projected/a11771b6-cb92-4783-bd70-4eb964d94f00-kube-api-access-p77tf\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:04.194351 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:04.163351 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n" event={"ID":"a11771b6-cb92-4783-bd70-4eb964d94f00","Type":"ContainerDied","Data":"0f9a4165f60665f39837aa72c376aec3dfe9a95077f5728b738c667756458269"} Oct 30 18:40:04.194351 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:04.163456 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n" Oct 30 18:40:04.194351 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:04.178968 4455 scope.go:122] "RemoveContainer" containerID="da55ebe22dd5589f70141c2b75d52cf3090a6b3b4a969324a1a3067c471199c9" Oct 30 18:40:04.356368 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:04.353317 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n"] Oct 30 18:40:04.390805 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:04.390738 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n"] Oct 30 18:40:05.867349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:05.866862 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a11771b6-cb92-4783-bd70-4eb964d94f00" path="/var/lib/kubelet/pods/a11771b6-cb92-4783-bd70-4eb964d94f00/volumes" Oct 30 18:40:06.057352 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:06.054484 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c8eef3-7b67449fc9-vhf9n" podUID="a11771b6-cb92-4783-bd70-4eb964d94f00" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.202:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:40:08.188142 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:08.188102 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266"] Oct 30 18:40:08.188803 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:08.188392 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266" podUID="4936cb40-0694-428e-bfd1-02a96a72a8a8" containerName="maas-api" containerID="cri-o://c720f5b81f703e660fbc315b474d5e3452ca5e9b9d63ce5864d993b6cbc8a60a" gracePeriod=30 Oct 30 18:40:08.346714 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:08.346684 4455 generic.go:356] "Generic (PLEG): container finished" podID="4936cb40-0694-428e-bfd1-02a96a72a8a8" containerID="c720f5b81f703e660fbc315b474d5e3452ca5e9b9d63ce5864d993b6cbc8a60a" exitCode=0 Oct 30 18:40:08.346896 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:08.346760 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266" event={"ID":"4936cb40-0694-428e-bfd1-02a96a72a8a8","Type":"ContainerDied","Data":"c720f5b81f703e660fbc315b474d5e3452ca5e9b9d63ce5864d993b6cbc8a60a"} Oct 30 18:40:08.751697 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:08.751671 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266" Oct 30 18:40:08.788519 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:08.788484 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4936cb40-0694-428e-bfd1-02a96a72a8a8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4936cb40-0694-428e-bfd1-02a96a72a8a8-openshift-service-ca\") pod \"4936cb40-0694-428e-bfd1-02a96a72a8a8\" (UID: \"4936cb40-0694-428e-bfd1-02a96a72a8a8\") " Oct 30 18:40:08.788713 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:08.788585 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4936cb40-0694-428e-bfd1-02a96a72a8a8-kube-api-access-qrgv7\" (UniqueName: \"kubernetes.io/projected/4936cb40-0694-428e-bfd1-02a96a72a8a8-kube-api-access-qrgv7\") pod \"4936cb40-0694-428e-bfd1-02a96a72a8a8\" (UID: \"4936cb40-0694-428e-bfd1-02a96a72a8a8\") " Oct 30 18:40:08.788789 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:08.788772 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4936cb40-0694-428e-bfd1-02a96a72a8a8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4936cb40-0694-428e-bfd1-02a96a72a8a8-maas-api-tls\") pod \"4936cb40-0694-428e-bfd1-02a96a72a8a8\" (UID: \"4936cb40-0694-428e-bfd1-02a96a72a8a8\") " Oct 30 18:40:08.788853 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:08.788829 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4936cb40-0694-428e-bfd1-02a96a72a8a8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4936cb40-0694-428e-bfd1-02a96a72a8a8-metrics-certs\") pod \"4936cb40-0694-428e-bfd1-02a96a72a8a8\" (UID: \"4936cb40-0694-428e-bfd1-02a96a72a8a8\") " Oct 30 18:40:08.788915 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:08.788881 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4936cb40-0694-428e-bfd1-02a96a72a8a8-openshift-service-ca" pod "4936cb40-0694-428e-bfd1-02a96a72a8a8" (UID: "4936cb40-0694-428e-bfd1-02a96a72a8a8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:08.789359 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:08.789309 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4936cb40-0694-428e-bfd1-02a96a72a8a8-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:08.791153 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:08.791124 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4936cb40-0694-428e-bfd1-02a96a72a8a8-maas-api-tls" pod "4936cb40-0694-428e-bfd1-02a96a72a8a8" (UID: "4936cb40-0694-428e-bfd1-02a96a72a8a8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:08.791300 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:08.791170 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4936cb40-0694-428e-bfd1-02a96a72a8a8-metrics-certs" pod "4936cb40-0694-428e-bfd1-02a96a72a8a8" (UID: "4936cb40-0694-428e-bfd1-02a96a72a8a8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:08.791501 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:08.791475 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4936cb40-0694-428e-bfd1-02a96a72a8a8-kube-api-access-qrgv7" pod "4936cb40-0694-428e-bfd1-02a96a72a8a8" (UID: "4936cb40-0694-428e-bfd1-02a96a72a8a8"). InnerVolumeSpecName "kube-api-access-qrgv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:08.891046 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:08.891014 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrgv7\" (UniqueName: \"kubernetes.io/projected/4936cb40-0694-428e-bfd1-02a96a72a8a8-kube-api-access-qrgv7\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:08.891046 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:08.891044 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4936cb40-0694-428e-bfd1-02a96a72a8a8-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:08.891046 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:08.891056 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4936cb40-0694-428e-bfd1-02a96a72a8a8-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:09.354858 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:09.354827 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266" Oct 30 18:40:09.354858 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:09.354854 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266" event={"ID":"4936cb40-0694-428e-bfd1-02a96a72a8a8","Type":"ContainerDied","Data":"c5babd4fa55ffebde173387fb6c2122394d45c6d8d8fcdcf5700a6a81d570e43"} Oct 30 18:40:09.355419 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:09.354915 4455 scope.go:122] "RemoveContainer" containerID="c720f5b81f703e660fbc315b474d5e3452ca5e9b9d63ce5864d993b6cbc8a60a" Oct 30 18:40:09.452717 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:09.452684 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266"] Oct 30 18:40:09.478757 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:09.478726 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-3b8bf3-7d69b6dd94-sd266"] Oct 30 18:40:09.635446 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:09.635363 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4936cb40-0694-428e-bfd1-02a96a72a8a8" path="/var/lib/kubelet/pods/4936cb40-0694-428e-bfd1-02a96a72a8a8/volumes" Oct 30 18:40:22.601992 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:22.601952 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3cc458-99565d668-p2qbm"] Oct 30 18:40:22.635674 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:22.635639 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3cc458-779ff4f8cf-lt8qr"] Oct 30 18:40:27.602705 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:27.602659 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc458-99565d668-p2qbm" podUID="c63830ed-816d-4e24-9900-7ee260bf622f" containerName="payload-processing" containerID="cri-o://4e6092f2a06e7cc0ce2d302edda308824e8a51731a79701cb6e067a4345a6f4a" gracePeriod=25 Oct 30 18:40:27.636393 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:27.636356 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3cc458-779ff4f8cf-lt8qr" podUID="2b13dbfe-f5db-46e6-810e-9d4ac231c94c" containerName="payload-pre-processing" containerID="cri-o://e95c10fcd9389a7ec3750097dcf42557ad184a631ad5c3a284dedfb38fdd2f2f" gracePeriod=25 Oct 30 18:40:29.533524 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:29.533491 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-419d32-6dff5c9996-jcbjp"] Oct 30 18:40:29.569921 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:29.569888 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-419d32-6b9bb548f9-bzxk2"] Oct 30 18:40:30.394066 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:40:30.393994 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:40:31.042078 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:31.042040 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3cc458-779ff4f8cf-lt8qr" podUID="2b13dbfe-f5db-46e6-810e-9d4ac231c94c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:40:31.042078 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:31.042059 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc458-99565d668-p2qbm" podUID="c63830ed-816d-4e24-9900-7ee260bf622f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:40:34.534304 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:34.534238 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-419d32-6dff5c9996-jcbjp" podUID="c1b171c5-abf2-4fd2-9b7f-5d103f44f704" containerName="payload-processing" containerID="cri-o://938d228428904ae56b628d23ca3afa1884eb384fea6ae20f0f67618c13456691" gracePeriod=25 Oct 30 18:40:34.570344 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:34.570282 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-419d32-6b9bb548f9-bzxk2" podUID="3f13fc8d-3df1-4f51-816e-69f10c2a03cb" containerName="payload-pre-processing" containerID="cri-o://d303bae62f2934ffa406d88baf6d08feb06d2de54509ae21486a6273c664155e" gracePeriod=25 Oct 30 18:40:35.926676 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:35.926638 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-419d32-6dff5c9996-jcbjp" podUID="c1b171c5-abf2-4fd2-9b7f-5d103f44f704" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:40:35.926676 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:35.926671 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-419d32-6b9bb548f9-bzxk2" podUID="3f13fc8d-3df1-4f51-816e-69f10c2a03cb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:40:41.042637 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:41.042584 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3cc458-779ff4f8cf-lt8qr" podUID="2b13dbfe-f5db-46e6-810e-9d4ac231c94c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:40:41.043156 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:41.042596 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc458-99565d668-p2qbm" podUID="c63830ed-816d-4e24-9900-7ee260bf622f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:40:45.927135 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:45.927090 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-419d32-6dff5c9996-jcbjp" podUID="c1b171c5-abf2-4fd2-9b7f-5d103f44f704" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:40:45.927617 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:45.927104 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-419d32-6b9bb548f9-bzxk2" podUID="3f13fc8d-3df1-4f51-816e-69f10c2a03cb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:40:46.829999 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:46.829946 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg"] Oct 30 18:40:46.832598 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:46.832558 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg" podUID="e62b028a-4ecb-46f8-9d6c-94bf236dc782" containerName="maas-api" containerID="cri-o://ef81a43bef31fccac078983d3c32dc95a61bcafc616372bd7d12f44baa117679" gracePeriod=30 Oct 30 18:40:47.305347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:47.296285 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg" Oct 30 18:40:47.359678 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:47.359646 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e62b028a-4ecb-46f8-9d6c-94bf236dc782-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e62b028a-4ecb-46f8-9d6c-94bf236dc782-maas-api-tls\") pod \"e62b028a-4ecb-46f8-9d6c-94bf236dc782\" (UID: \"e62b028a-4ecb-46f8-9d6c-94bf236dc782\") " Oct 30 18:40:47.359878 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:47.359736 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e62b028a-4ecb-46f8-9d6c-94bf236dc782-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e62b028a-4ecb-46f8-9d6c-94bf236dc782-metrics-certs\") pod \"e62b028a-4ecb-46f8-9d6c-94bf236dc782\" (UID: \"e62b028a-4ecb-46f8-9d6c-94bf236dc782\") " Oct 30 18:40:47.359878 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:47.359791 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e62b028a-4ecb-46f8-9d6c-94bf236dc782-kube-api-access-hg6lf\" (UniqueName: \"kubernetes.io/projected/e62b028a-4ecb-46f8-9d6c-94bf236dc782-kube-api-access-hg6lf\") pod \"e62b028a-4ecb-46f8-9d6c-94bf236dc782\" (UID: \"e62b028a-4ecb-46f8-9d6c-94bf236dc782\") " Oct 30 18:40:47.359878 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:47.359838 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e62b028a-4ecb-46f8-9d6c-94bf236dc782-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e62b028a-4ecb-46f8-9d6c-94bf236dc782-openshift-service-ca\") pod \"e62b028a-4ecb-46f8-9d6c-94bf236dc782\" (UID: \"e62b028a-4ecb-46f8-9d6c-94bf236dc782\") " Oct 30 18:40:47.360699 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:47.360668 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e62b028a-4ecb-46f8-9d6c-94bf236dc782-openshift-service-ca" pod "e62b028a-4ecb-46f8-9d6c-94bf236dc782" (UID: "e62b028a-4ecb-46f8-9d6c-94bf236dc782"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:47.366492 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:47.366456 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e62b028a-4ecb-46f8-9d6c-94bf236dc782-metrics-certs" pod "e62b028a-4ecb-46f8-9d6c-94bf236dc782" (UID: "e62b028a-4ecb-46f8-9d6c-94bf236dc782"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:47.366735 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:47.366707 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e62b028a-4ecb-46f8-9d6c-94bf236dc782-kube-api-access-hg6lf" pod "e62b028a-4ecb-46f8-9d6c-94bf236dc782" (UID: "e62b028a-4ecb-46f8-9d6c-94bf236dc782"). InnerVolumeSpecName "kube-api-access-hg6lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:47.386913 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:47.386852 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e62b028a-4ecb-46f8-9d6c-94bf236dc782-maas-api-tls" pod "e62b028a-4ecb-46f8-9d6c-94bf236dc782" (UID: "e62b028a-4ecb-46f8-9d6c-94bf236dc782"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:47.463584 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:47.463478 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e62b028a-4ecb-46f8-9d6c-94bf236dc782-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:47.463584 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:47.463518 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e62b028a-4ecb-46f8-9d6c-94bf236dc782-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:47.463584 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:47.463535 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg6lf\" (UniqueName: \"kubernetes.io/projected/e62b028a-4ecb-46f8-9d6c-94bf236dc782-kube-api-access-hg6lf\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:47.463584 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:47.463551 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e62b028a-4ecb-46f8-9d6c-94bf236dc782-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:47.735698 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:47.735662 4455 generic.go:356] "Generic (PLEG): container finished" podID="c63830ed-816d-4e24-9900-7ee260bf622f" containerID="4e6092f2a06e7cc0ce2d302edda308824e8a51731a79701cb6e067a4345a6f4a" exitCode=0 Oct 30 18:40:47.735886 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:47.735737 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc458-99565d668-p2qbm" event={"ID":"c63830ed-816d-4e24-9900-7ee260bf622f","Type":"ContainerDied","Data":"4e6092f2a06e7cc0ce2d302edda308824e8a51731a79701cb6e067a4345a6f4a"} Oct 30 18:40:47.737633 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:47.737609 4455 generic.go:356] "Generic (PLEG): container finished" podID="e62b028a-4ecb-46f8-9d6c-94bf236dc782" containerID="ef81a43bef31fccac078983d3c32dc95a61bcafc616372bd7d12f44baa117679" exitCode=0 Oct 30 18:40:47.737751 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:47.737657 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg" Oct 30 18:40:47.737751 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:47.737665 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg" event={"ID":"e62b028a-4ecb-46f8-9d6c-94bf236dc782","Type":"ContainerDied","Data":"ef81a43bef31fccac078983d3c32dc95a61bcafc616372bd7d12f44baa117679"} Oct 30 18:40:47.737751 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:47.737702 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg" event={"ID":"e62b028a-4ecb-46f8-9d6c-94bf236dc782","Type":"ContainerDied","Data":"ed19c24b68aa163cd7e3dbb5460461a3486bef7d2435eb6fd36a00b7aebe3a8a"} Oct 30 18:40:47.737751 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:47.737724 4455 scope.go:122] "RemoveContainer" containerID="ef81a43bef31fccac078983d3c32dc95a61bcafc616372bd7d12f44baa117679" Oct 30 18:40:47.750910 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:47.750624 4455 scope.go:122] "RemoveContainer" containerID="98115b037cfee465085dfba99e933e44ef8c1d9dff7ca6329f789a07af8f720d" Oct 30 18:40:47.762038 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:47.762017 4455 scope.go:122] "RemoveContainer" containerID="ef81a43bef31fccac078983d3c32dc95a61bcafc616372bd7d12f44baa117679" Oct 30 18:40:47.762478 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:40:47.762437 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef81a43bef31fccac078983d3c32dc95a61bcafc616372bd7d12f44baa117679\": container with ID starting with ef81a43bef31fccac078983d3c32dc95a61bcafc616372bd7d12f44baa117679 not found: ID does not exist" containerID="ef81a43bef31fccac078983d3c32dc95a61bcafc616372bd7d12f44baa117679" Oct 30 18:40:47.762558 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:47.762498 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef81a43bef31fccac078983d3c32dc95a61bcafc616372bd7d12f44baa117679"} err="failed to get container status \"ef81a43bef31fccac078983d3c32dc95a61bcafc616372bd7d12f44baa117679\": rpc error: code = NotFound desc = could not find container \"ef81a43bef31fccac078983d3c32dc95a61bcafc616372bd7d12f44baa117679\": container with ID starting with ef81a43bef31fccac078983d3c32dc95a61bcafc616372bd7d12f44baa117679 not found: ID does not exist" Oct 30 18:40:47.762558 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:47.762531 4455 scope.go:122] "RemoveContainer" containerID="98115b037cfee465085dfba99e933e44ef8c1d9dff7ca6329f789a07af8f720d" Oct 30 18:40:47.762946 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:40:47.762908 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98115b037cfee465085dfba99e933e44ef8c1d9dff7ca6329f789a07af8f720d\": container with ID starting with 98115b037cfee465085dfba99e933e44ef8c1d9dff7ca6329f789a07af8f720d not found: ID does not exist" containerID="98115b037cfee465085dfba99e933e44ef8c1d9dff7ca6329f789a07af8f720d" Oct 30 18:40:47.763022 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:47.762944 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98115b037cfee465085dfba99e933e44ef8c1d9dff7ca6329f789a07af8f720d"} err="failed to get container status \"98115b037cfee465085dfba99e933e44ef8c1d9dff7ca6329f789a07af8f720d\": rpc error: code = NotFound desc = could not find container \"98115b037cfee465085dfba99e933e44ef8c1d9dff7ca6329f789a07af8f720d\": container with ID starting with 98115b037cfee465085dfba99e933e44ef8c1d9dff7ca6329f789a07af8f720d not found: ID does not exist" Oct 30 18:40:47.804354 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:47.801583 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg"] Oct 30 18:40:47.829348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:47.829004 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-419d32-75dc876d85-ht9rg"] Oct 30 18:40:48.000945 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:48.000877 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc458-99565d668-p2qbm" Oct 30 18:40:48.062776 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:48.062343 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3cc458-779ff4f8cf-lt8qr" Oct 30 18:40:48.099812 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:48.099778 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c63830ed-816d-4e24-9900-7ee260bf622f-kube-api-access-5rsrm\" (UniqueName: \"kubernetes.io/projected/c63830ed-816d-4e24-9900-7ee260bf622f-kube-api-access-5rsrm\") pod \"c63830ed-816d-4e24-9900-7ee260bf622f\" (UID: \"c63830ed-816d-4e24-9900-7ee260bf622f\") " Oct 30 18:40:48.100042 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:48.100024 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c63830ed-816d-4e24-9900-7ee260bf622f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c63830ed-816d-4e24-9900-7ee260bf622f-plugins-config-volume\") pod \"c63830ed-816d-4e24-9900-7ee260bf622f\" (UID: \"c63830ed-816d-4e24-9900-7ee260bf622f\") " Oct 30 18:40:48.100598 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:48.100567 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c63830ed-816d-4e24-9900-7ee260bf622f-plugins-config-volume" pod "c63830ed-816d-4e24-9900-7ee260bf622f" (UID: "c63830ed-816d-4e24-9900-7ee260bf622f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:48.102191 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:48.102168 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c63830ed-816d-4e24-9900-7ee260bf622f-kube-api-access-5rsrm" pod "c63830ed-816d-4e24-9900-7ee260bf622f" (UID: "c63830ed-816d-4e24-9900-7ee260bf622f"). InnerVolumeSpecName "kube-api-access-5rsrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:48.201292 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:48.201259 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b13dbfe-f5db-46e6-810e-9d4ac231c94c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b13dbfe-f5db-46e6-810e-9d4ac231c94c-plugins-config-volume\") pod \"2b13dbfe-f5db-46e6-810e-9d4ac231c94c\" (UID: \"2b13dbfe-f5db-46e6-810e-9d4ac231c94c\") " Oct 30 18:40:48.201506 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:48.201384 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b13dbfe-f5db-46e6-810e-9d4ac231c94c-kube-api-access-khnrs\" (UniqueName: \"kubernetes.io/projected/2b13dbfe-f5db-46e6-810e-9d4ac231c94c-kube-api-access-khnrs\") pod \"2b13dbfe-f5db-46e6-810e-9d4ac231c94c\" (UID: \"2b13dbfe-f5db-46e6-810e-9d4ac231c94c\") " Oct 30 18:40:48.201743 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:48.201719 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b13dbfe-f5db-46e6-810e-9d4ac231c94c-plugins-config-volume" pod "2b13dbfe-f5db-46e6-810e-9d4ac231c94c" (UID: "2b13dbfe-f5db-46e6-810e-9d4ac231c94c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:48.201990 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:48.201976 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c63830ed-816d-4e24-9900-7ee260bf622f-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:48.202040 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:48.201995 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rsrm\" (UniqueName: \"kubernetes.io/projected/c63830ed-816d-4e24-9900-7ee260bf622f-kube-api-access-5rsrm\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:48.202040 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:48.202008 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b13dbfe-f5db-46e6-810e-9d4ac231c94c-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:48.203684 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:48.203661 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b13dbfe-f5db-46e6-810e-9d4ac231c94c-kube-api-access-khnrs" pod "2b13dbfe-f5db-46e6-810e-9d4ac231c94c" (UID: "2b13dbfe-f5db-46e6-810e-9d4ac231c94c"). InnerVolumeSpecName "kube-api-access-khnrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:48.304220 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:48.304138 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khnrs\" (UniqueName: \"kubernetes.io/projected/2b13dbfe-f5db-46e6-810e-9d4ac231c94c-kube-api-access-khnrs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:48.747293 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:48.747254 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc458-99565d668-p2qbm" Oct 30 18:40:48.747293 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:48.747276 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc458-99565d668-p2qbm" event={"ID":"c63830ed-816d-4e24-9900-7ee260bf622f","Type":"ContainerDied","Data":"be3922a24dd3acc13abead6fb78648ee594d834ead59fe02eeeec86ce235ac3b"} Oct 30 18:40:48.747878 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:48.747388 4455 scope.go:122] "RemoveContainer" containerID="4e6092f2a06e7cc0ce2d302edda308824e8a51731a79701cb6e067a4345a6f4a" Oct 30 18:40:48.750221 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:48.750195 4455 generic.go:356] "Generic (PLEG): container finished" podID="2b13dbfe-f5db-46e6-810e-9d4ac231c94c" containerID="e95c10fcd9389a7ec3750097dcf42557ad184a631ad5c3a284dedfb38fdd2f2f" exitCode=0 Oct 30 18:40:48.750346 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:48.750297 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3cc458-779ff4f8cf-lt8qr" event={"ID":"2b13dbfe-f5db-46e6-810e-9d4ac231c94c","Type":"ContainerDied","Data":"e95c10fcd9389a7ec3750097dcf42557ad184a631ad5c3a284dedfb38fdd2f2f"} Oct 30 18:40:48.750401 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:48.750373 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3cc458-779ff4f8cf-lt8qr" event={"ID":"2b13dbfe-f5db-46e6-810e-9d4ac231c94c","Type":"ContainerDied","Data":"2740a3c47cf21fdc091893f2b137d22e8da4e3a92ce92149871138c2b2bc5da9"} Oct 30 18:40:48.750465 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:48.750453 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3cc458-779ff4f8cf-lt8qr" Oct 30 18:40:48.758928 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:48.758871 4455 scope.go:122] "RemoveContainer" containerID="e95c10fcd9389a7ec3750097dcf42557ad184a631ad5c3a284dedfb38fdd2f2f" Oct 30 18:40:48.768980 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:48.768959 4455 scope.go:122] "RemoveContainer" containerID="e95c10fcd9389a7ec3750097dcf42557ad184a631ad5c3a284dedfb38fdd2f2f" Oct 30 18:40:48.769262 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:40:48.769238 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e95c10fcd9389a7ec3750097dcf42557ad184a631ad5c3a284dedfb38fdd2f2f\": container with ID starting with e95c10fcd9389a7ec3750097dcf42557ad184a631ad5c3a284dedfb38fdd2f2f not found: ID does not exist" containerID="e95c10fcd9389a7ec3750097dcf42557ad184a631ad5c3a284dedfb38fdd2f2f" Oct 30 18:40:48.769353 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:48.769271 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e95c10fcd9389a7ec3750097dcf42557ad184a631ad5c3a284dedfb38fdd2f2f"} err="failed to get container status \"e95c10fcd9389a7ec3750097dcf42557ad184a631ad5c3a284dedfb38fdd2f2f\": rpc error: code = NotFound desc = could not find container \"e95c10fcd9389a7ec3750097dcf42557ad184a631ad5c3a284dedfb38fdd2f2f\": container with ID starting with e95c10fcd9389a7ec3750097dcf42557ad184a631ad5c3a284dedfb38fdd2f2f not found: ID does not exist" Oct 30 18:40:48.799045 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:48.799015 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3cc458-99565d668-p2qbm"] Oct 30 18:40:48.810196 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:48.810163 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3cc458-99565d668-p2qbm"] Oct 30 18:40:48.843282 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:48.843249 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3cc458-779ff4f8cf-lt8qr"] Oct 30 18:40:48.848441 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:48.848414 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3cc458-779ff4f8cf-lt8qr"] Oct 30 18:40:49.634971 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:49.634931 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b13dbfe-f5db-46e6-810e-9d4ac231c94c" path="/var/lib/kubelet/pods/2b13dbfe-f5db-46e6-810e-9d4ac231c94c/volumes" Oct 30 18:40:49.635385 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:49.635370 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c63830ed-816d-4e24-9900-7ee260bf622f" path="/var/lib/kubelet/pods/c63830ed-816d-4e24-9900-7ee260bf622f/volumes" Oct 30 18:40:49.635756 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:49.635742 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e62b028a-4ecb-46f8-9d6c-94bf236dc782" path="/var/lib/kubelet/pods/e62b028a-4ecb-46f8-9d6c-94bf236dc782/volumes" Oct 30 18:40:50.228386 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:50.225010 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg"] Oct 30 18:40:50.228386 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:50.225264 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" podUID="70d7e6c8-8d0f-40f0-a533-58158542fb7c" containerName="istio-proxy" containerID="cri-o://80af84e433f2921c01f3cb8069dc5888444f4e02f28294a6f2f627170a368f6f" gracePeriod=30 Oct 30 18:40:50.251645 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:50.248639 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s"] Oct 30 18:40:50.251645 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:50.248919 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" podUID="d4e8e077-c9f7-44c8-9dd5-d1c7a2403862" containerName="istio-proxy" containerID="cri-o://909682ad7c983b900fe1e1f104fd35ebac75d470a756e2922f03fdbae688d41f" gracePeriod=30 Oct 30 18:40:50.257586 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:50.257561 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584"] Oct 30 18:40:50.258019 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:50.257987 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" podUID="d7de539f-17e9-453c-af6b-e83fc7c91239" containerName="istio-proxy" containerID="cri-o://4c0afb4f15150e31f8a16b39c86d19d0b16977cddbc40c5b5cf4debe0e59c567" gracePeriod=30 Oct 30 18:40:54.830439 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:54.830163 4455 generic.go:356] "Generic (PLEG): container finished" podID="3f13fc8d-3df1-4f51-816e-69f10c2a03cb" containerID="d303bae62f2934ffa406d88baf6d08feb06d2de54509ae21486a6273c664155e" exitCode=0 Oct 30 18:40:54.830439 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:54.830350 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-419d32-6b9bb548f9-bzxk2" event={"ID":"3f13fc8d-3df1-4f51-816e-69f10c2a03cb","Type":"ContainerDied","Data":"d303bae62f2934ffa406d88baf6d08feb06d2de54509ae21486a6273c664155e"} Oct 30 18:40:54.833063 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:54.833029 4455 generic.go:356] "Generic (PLEG): container finished" podID="c1b171c5-abf2-4fd2-9b7f-5d103f44f704" containerID="938d228428904ae56b628d23ca3afa1884eb384fea6ae20f0f67618c13456691" exitCode=0 Oct 30 18:40:54.833210 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:54.833134 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-419d32-6dff5c9996-jcbjp" event={"ID":"c1b171c5-abf2-4fd2-9b7f-5d103f44f704","Type":"ContainerDied","Data":"938d228428904ae56b628d23ca3afa1884eb384fea6ae20f0f67618c13456691"} Oct 30 18:40:54.960536 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:54.960508 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-419d32-6b9bb548f9-bzxk2" Oct 30 18:40:54.986893 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:54.986861 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-419d32-6dff5c9996-jcbjp" Oct 30 18:40:55.174144 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.174055 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f13fc8d-3df1-4f51-816e-69f10c2a03cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f13fc8d-3df1-4f51-816e-69f10c2a03cb-plugins-config-volume\") pod \"3f13fc8d-3df1-4f51-816e-69f10c2a03cb\" (UID: \"3f13fc8d-3df1-4f51-816e-69f10c2a03cb\") " Oct 30 18:40:55.174443 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.174422 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1b171c5-abf2-4fd2-9b7f-5d103f44f704-kube-api-access-pnllz\" (UniqueName: \"kubernetes.io/projected/c1b171c5-abf2-4fd2-9b7f-5d103f44f704-kube-api-access-pnllz\") pod \"c1b171c5-abf2-4fd2-9b7f-5d103f44f704\" (UID: \"c1b171c5-abf2-4fd2-9b7f-5d103f44f704\") " Oct 30 18:40:55.174669 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.174654 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f13fc8d-3df1-4f51-816e-69f10c2a03cb-kube-api-access-gw6cp\" (UniqueName: \"kubernetes.io/projected/3f13fc8d-3df1-4f51-816e-69f10c2a03cb-kube-api-access-gw6cp\") pod \"3f13fc8d-3df1-4f51-816e-69f10c2a03cb\" (UID: \"3f13fc8d-3df1-4f51-816e-69f10c2a03cb\") " Oct 30 18:40:55.174873 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.174859 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1b171c5-abf2-4fd2-9b7f-5d103f44f704-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b171c5-abf2-4fd2-9b7f-5d103f44f704-plugins-config-volume\") pod \"c1b171c5-abf2-4fd2-9b7f-5d103f44f704\" (UID: \"c1b171c5-abf2-4fd2-9b7f-5d103f44f704\") " Oct 30 18:40:55.177377 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.177309 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f13fc8d-3df1-4f51-816e-69f10c2a03cb-plugins-config-volume" pod "3f13fc8d-3df1-4f51-816e-69f10c2a03cb" (UID: "3f13fc8d-3df1-4f51-816e-69f10c2a03cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:55.177847 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.176010 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1b171c5-abf2-4fd2-9b7f-5d103f44f704-plugins-config-volume" pod "c1b171c5-abf2-4fd2-9b7f-5d103f44f704" (UID: "c1b171c5-abf2-4fd2-9b7f-5d103f44f704"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:55.196434 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.190147 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f13fc8d-3df1-4f51-816e-69f10c2a03cb-kube-api-access-gw6cp" pod "3f13fc8d-3df1-4f51-816e-69f10c2a03cb" (UID: "3f13fc8d-3df1-4f51-816e-69f10c2a03cb"). InnerVolumeSpecName "kube-api-access-gw6cp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:55.201574 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.201539 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b171c5-abf2-4fd2-9b7f-5d103f44f704-kube-api-access-pnllz" pod "c1b171c5-abf2-4fd2-9b7f-5d103f44f704" (UID: "c1b171c5-abf2-4fd2-9b7f-5d103f44f704"). InnerVolumeSpecName "kube-api-access-pnllz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:55.276518 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.276480 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b171c5-abf2-4fd2-9b7f-5d103f44f704-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:55.276518 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.276516 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f13fc8d-3df1-4f51-816e-69f10c2a03cb-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:55.276751 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.276528 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnllz\" (UniqueName: \"kubernetes.io/projected/c1b171c5-abf2-4fd2-9b7f-5d103f44f704-kube-api-access-pnllz\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:55.276751 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.276543 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw6cp\" (UniqueName: \"kubernetes.io/projected/3f13fc8d-3df1-4f51-816e-69f10c2a03cb-kube-api-access-gw6cp\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:55.620174 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.619869 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" Oct 30 18:40:55.823103 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.796091 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70d7e6c8-8d0f-40f0-a533-58158542fb7c-istio-token\" (UniqueName: \"kubernetes.io/projected/70d7e6c8-8d0f-40f0-a533-58158542fb7c-istio-token\") pod \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " Oct 30 18:40:55.823103 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.796195 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70d7e6c8-8d0f-40f0-a533-58158542fb7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70d7e6c8-8d0f-40f0-a533-58158542fb7c-istio-envoy\") pod \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " Oct 30 18:40:55.823103 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.796237 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70d7e6c8-8d0f-40f0-a533-58158542fb7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d7e6c8-8d0f-40f0-a533-58158542fb7c-credential-socket\") pod \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " Oct 30 18:40:55.823103 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.796276 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70d7e6c8-8d0f-40f0-a533-58158542fb7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70d7e6c8-8d0f-40f0-a533-58158542fb7c-istio-data\") pod \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " Oct 30 18:40:55.823103 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.796350 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70d7e6c8-8d0f-40f0-a533-58158542fb7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70d7e6c8-8d0f-40f0-a533-58158542fb7c-istiod-ca-cert\") pod \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " Oct 30 18:40:55.823103 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.796398 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70d7e6c8-8d0f-40f0-a533-58158542fb7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70d7e6c8-8d0f-40f0-a533-58158542fb7c-istio-podinfo\") pod \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " Oct 30 18:40:55.823103 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.796456 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70d7e6c8-8d0f-40f0-a533-58158542fb7c-kube-api-access-92vhn\" (UniqueName: \"kubernetes.io/projected/70d7e6c8-8d0f-40f0-a533-58158542fb7c-kube-api-access-92vhn\") pod \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " Oct 30 18:40:55.823103 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.796555 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70d7e6c8-8d0f-40f0-a533-58158542fb7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d7e6c8-8d0f-40f0-a533-58158542fb7c-workload-socket\") pod \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " Oct 30 18:40:55.823103 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.796657 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70d7e6c8-8d0f-40f0-a533-58158542fb7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70d7e6c8-8d0f-40f0-a533-58158542fb7c-workload-certs\") pod \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\" (UID: \"70d7e6c8-8d0f-40f0-a533-58158542fb7c\") " Oct 30 18:40:55.828651 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.827411 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70d7e6c8-8d0f-40f0-a533-58158542fb7c-istio-data" pod "70d7e6c8-8d0f-40f0-a533-58158542fb7c" (UID: "70d7e6c8-8d0f-40f0-a533-58158542fb7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:55.828651 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.828260 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70d7e6c8-8d0f-40f0-a533-58158542fb7c-workload-socket" pod "70d7e6c8-8d0f-40f0-a533-58158542fb7c" (UID: "70d7e6c8-8d0f-40f0-a533-58158542fb7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:55.828651 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.828542 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70d7e6c8-8d0f-40f0-a533-58158542fb7c-istiod-ca-cert" pod "70d7e6c8-8d0f-40f0-a533-58158542fb7c" (UID: "70d7e6c8-8d0f-40f0-a533-58158542fb7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:55.828651 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.828612 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70d7e6c8-8d0f-40f0-a533-58158542fb7c-credential-socket" pod "70d7e6c8-8d0f-40f0-a533-58158542fb7c" (UID: "70d7e6c8-8d0f-40f0-a533-58158542fb7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:55.829403 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.829362 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70d7e6c8-8d0f-40f0-a533-58158542fb7c-workload-certs" pod "70d7e6c8-8d0f-40f0-a533-58158542fb7c" (UID: "70d7e6c8-8d0f-40f0-a533-58158542fb7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:55.879728 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.860894 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-419d32-6b9bb548f9-bzxk2" event={"ID":"3f13fc8d-3df1-4f51-816e-69f10c2a03cb","Type":"ContainerDied","Data":"429ad68aa945efe94999881158b02d77b2b6935f1579d7999739cb7bd528c644"} Oct 30 18:40:55.879728 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.860988 4455 scope.go:122] "RemoveContainer" containerID="d303bae62f2934ffa406d88baf6d08feb06d2de54509ae21486a6273c664155e" Oct 30 18:40:55.879728 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.861180 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-419d32-6b9bb548f9-bzxk2" Oct 30 18:40:55.879728 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.873222 4455 generic.go:356] "Generic (PLEG): container finished" podID="70d7e6c8-8d0f-40f0-a533-58158542fb7c" containerID="80af84e433f2921c01f3cb8069dc5888444f4e02f28294a6f2f627170a368f6f" exitCode=0 Oct 30 18:40:55.879728 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.873486 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" event={"ID":"70d7e6c8-8d0f-40f0-a533-58158542fb7c","Type":"ContainerDied","Data":"80af84e433f2921c01f3cb8069dc5888444f4e02f28294a6f2f627170a368f6f"} Oct 30 18:40:55.879728 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.873701 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" Oct 30 18:40:55.960388 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.891295 4455 generic.go:356] "Generic (PLEG): container finished" podID="d7de539f-17e9-453c-af6b-e83fc7c91239" containerID="4c0afb4f15150e31f8a16b39c86d19d0b16977cddbc40c5b5cf4debe0e59c567" exitCode=0 Oct 30 18:40:55.960388 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.913270 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-419d32-6dff5c9996-jcbjp" Oct 30 18:40:55.960388 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.921428 4455 generic.go:356] "Generic (PLEG): container finished" podID="d4e8e077-c9f7-44c8-9dd5-d1c7a2403862" containerID="909682ad7c983b900fe1e1f104fd35ebac75d470a756e2922f03fdbae688d41f" exitCode=0 Oct 30 18:40:55.960388 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.921956 4455 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d7e6c8-8d0f-40f0-a533-58158542fb7c-credential-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:55.960388 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.921988 4455 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70d7e6c8-8d0f-40f0-a533-58158542fb7c-istio-data\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:55.960388 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.922007 4455 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70d7e6c8-8d0f-40f0-a533-58158542fb7c-istiod-ca-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:55.960388 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.922019 4455 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d7e6c8-8d0f-40f0-a533-58158542fb7c-workload-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:55.960388 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.922032 4455 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70d7e6c8-8d0f-40f0-a533-58158542fb7c-workload-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:55.973363 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.950894 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg" event={"ID":"70d7e6c8-8d0f-40f0-a533-58158542fb7c","Type":"ContainerDied","Data":"d4405654400a6047c17679af6e61c2bcc5cd8b0a8bb8daec54431afd53fbba4a"} Oct 30 18:40:55.973363 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.962267 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" event={"ID":"d7de539f-17e9-453c-af6b-e83fc7c91239","Type":"ContainerDied","Data":"4c0afb4f15150e31f8a16b39c86d19d0b16977cddbc40c5b5cf4debe0e59c567"} Oct 30 18:40:55.973363 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.962302 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-419d32-6dff5c9996-jcbjp" event={"ID":"c1b171c5-abf2-4fd2-9b7f-5d103f44f704","Type":"ContainerDied","Data":"450a3b3f13ac05b3ff09efbe3def600495d31b1a8b328e15c6e73a2cd82d0132"} Oct 30 18:40:55.973363 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.962341 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" event={"ID":"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862","Type":"ContainerDied","Data":"909682ad7c983b900fe1e1f104fd35ebac75d470a756e2922f03fdbae688d41f"} Oct 30 18:40:55.994361 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:55.992166 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-419d32-6b9bb548f9-bzxk2"] Oct 30 18:40:56.044743 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.044507 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70d7e6c8-8d0f-40f0-a533-58158542fb7c-istio-token" pod "70d7e6c8-8d0f-40f0-a533-58158542fb7c" (UID: "70d7e6c8-8d0f-40f0-a533-58158542fb7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:56.044929 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.044810 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-419d32-6b9bb548f9-bzxk2"] Oct 30 18:40:56.053814 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.052491 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70d7e6c8-8d0f-40f0-a533-58158542fb7c-istio-envoy" pod "70d7e6c8-8d0f-40f0-a533-58158542fb7c" (UID: "70d7e6c8-8d0f-40f0-a533-58158542fb7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:56.054112 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.054069 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70d7e6c8-8d0f-40f0-a533-58158542fb7c-kube-api-access-92vhn" pod "70d7e6c8-8d0f-40f0-a533-58158542fb7c" (UID: "70d7e6c8-8d0f-40f0-a533-58158542fb7c"). InnerVolumeSpecName "kube-api-access-92vhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:56.119220 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.118512 4455 scope.go:122] "RemoveContainer" containerID="80af84e433f2921c01f3cb8069dc5888444f4e02f28294a6f2f627170a368f6f" Oct 30 18:40:56.130649 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.123531 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70d7e6c8-8d0f-40f0-a533-58158542fb7c-istio-podinfo" pod "70d7e6c8-8d0f-40f0-a533-58158542fb7c" (UID: "70d7e6c8-8d0f-40f0-a533-58158542fb7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:40:56.143581 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.139459 4455 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70d7e6c8-8d0f-40f0-a533-58158542fb7c-istio-token\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.143581 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.139485 4455 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70d7e6c8-8d0f-40f0-a533-58158542fb7c-istio-envoy\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.143581 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.139500 4455 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70d7e6c8-8d0f-40f0-a533-58158542fb7c-istio-podinfo\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.143581 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.139514 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92vhn\" (UniqueName: \"kubernetes.io/projected/70d7e6c8-8d0f-40f0-a533-58158542fb7c-kube-api-access-92vhn\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.164839 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.164527 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" Oct 30 18:40:56.174433 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.174360 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-419d32-6dff5c9996-jcbjp"] Oct 30 18:40:56.349944 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.348502 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-istio-token\" (UniqueName: \"kubernetes.io/projected/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-istio-token\") pod \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " Oct 30 18:40:56.349944 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.348564 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-istio-data\") pod \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " Oct 30 18:40:56.349944 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.348638 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-kube-api-access-g2m4n\" (UniqueName: \"kubernetes.io/projected/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-kube-api-access-g2m4n\") pod \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " Oct 30 18:40:56.349944 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.348667 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-workload-socket\") pod \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " Oct 30 18:40:56.349944 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.348721 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-istio-envoy\") pod \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " Oct 30 18:40:56.349944 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.348802 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-istiod-ca-cert\") pod \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " Oct 30 18:40:56.349944 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.348868 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-credential-socket\") pod \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " Oct 30 18:40:56.349944 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.348979 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-workload-certs\") pod \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " Oct 30 18:40:56.349944 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.349034 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-istio-podinfo\") pod \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\" (UID: \"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862\") " Oct 30 18:40:56.389828 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.389434 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-workload-socket" pod "d4e8e077-c9f7-44c8-9dd5-d1c7a2403862" (UID: "d4e8e077-c9f7-44c8-9dd5-d1c7a2403862"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:56.406442 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.390982 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-credential-socket" pod "d4e8e077-c9f7-44c8-9dd5-d1c7a2403862" (UID: "d4e8e077-c9f7-44c8-9dd5-d1c7a2403862"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:56.406442 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.391352 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-istiod-ca-cert" pod "d4e8e077-c9f7-44c8-9dd5-d1c7a2403862" (UID: "d4e8e077-c9f7-44c8-9dd5-d1c7a2403862"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:56.406442 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.391625 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-workload-certs" pod "d4e8e077-c9f7-44c8-9dd5-d1c7a2403862" (UID: "d4e8e077-c9f7-44c8-9dd5-d1c7a2403862"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:56.406442 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.393486 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" Oct 30 18:40:56.407058 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.406567 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-istio-data" pod "d4e8e077-c9f7-44c8-9dd5-d1c7a2403862" (UID: "d4e8e077-c9f7-44c8-9dd5-d1c7a2403862"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:56.454972 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.451265 4455 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-workload-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.454972 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.451300 4455 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-istio-data\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.475390 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.451315 4455 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-workload-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.475390 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.470682 4455 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-istiod-ca-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.475390 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.470702 4455 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-credential-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.569367 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.532483 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-istio-envoy" pod "d4e8e077-c9f7-44c8-9dd5-d1c7a2403862" (UID: "d4e8e077-c9f7-44c8-9dd5-d1c7a2403862"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:56.578987 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.573014 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7de539f-17e9-453c-af6b-e83fc7c91239-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7de539f-17e9-453c-af6b-e83fc7c91239-workload-certs\") pod \"d7de539f-17e9-453c-af6b-e83fc7c91239\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " Oct 30 18:40:56.578987 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.573099 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7de539f-17e9-453c-af6b-e83fc7c91239-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7de539f-17e9-453c-af6b-e83fc7c91239-istio-envoy\") pod \"d7de539f-17e9-453c-af6b-e83fc7c91239\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " Oct 30 18:40:56.578987 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.573142 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7de539f-17e9-453c-af6b-e83fc7c91239-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7de539f-17e9-453c-af6b-e83fc7c91239-workload-socket\") pod \"d7de539f-17e9-453c-af6b-e83fc7c91239\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " Oct 30 18:40:56.578987 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.573214 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7de539f-17e9-453c-af6b-e83fc7c91239-istio-token\" (UniqueName: \"kubernetes.io/projected/d7de539f-17e9-453c-af6b-e83fc7c91239-istio-token\") pod \"d7de539f-17e9-453c-af6b-e83fc7c91239\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " Oct 30 18:40:56.578987 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.573258 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7de539f-17e9-453c-af6b-e83fc7c91239-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7de539f-17e9-453c-af6b-e83fc7c91239-credential-socket\") pod \"d7de539f-17e9-453c-af6b-e83fc7c91239\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " Oct 30 18:40:56.595517 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.586590 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7de539f-17e9-453c-af6b-e83fc7c91239-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7de539f-17e9-453c-af6b-e83fc7c91239-istiod-ca-cert\") pod \"d7de539f-17e9-453c-af6b-e83fc7c91239\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " Oct 30 18:40:56.595517 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.586646 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7de539f-17e9-453c-af6b-e83fc7c91239-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7de539f-17e9-453c-af6b-e83fc7c91239-istio-data\") pod \"d7de539f-17e9-453c-af6b-e83fc7c91239\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " Oct 30 18:40:56.595517 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.586732 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7de539f-17e9-453c-af6b-e83fc7c91239-kube-api-access-jqbm6\" (UniqueName: \"kubernetes.io/projected/d7de539f-17e9-453c-af6b-e83fc7c91239-kube-api-access-jqbm6\") pod \"d7de539f-17e9-453c-af6b-e83fc7c91239\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " Oct 30 18:40:56.595517 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.586823 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7de539f-17e9-453c-af6b-e83fc7c91239-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7de539f-17e9-453c-af6b-e83fc7c91239-istio-podinfo\") pod \"d7de539f-17e9-453c-af6b-e83fc7c91239\" (UID: \"d7de539f-17e9-453c-af6b-e83fc7c91239\") " Oct 30 18:40:56.595517 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.587443 4455 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-istio-envoy\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.595517 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.587734 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7de539f-17e9-453c-af6b-e83fc7c91239-workload-certs" pod "d7de539f-17e9-453c-af6b-e83fc7c91239" (UID: "d7de539f-17e9-453c-af6b-e83fc7c91239"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:56.623393 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.601166 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7de539f-17e9-453c-af6b-e83fc7c91239-credential-socket" pod "d7de539f-17e9-453c-af6b-e83fc7c91239" (UID: "d7de539f-17e9-453c-af6b-e83fc7c91239"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:56.623393 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.616545 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7de539f-17e9-453c-af6b-e83fc7c91239-istiod-ca-cert" pod "d7de539f-17e9-453c-af6b-e83fc7c91239" (UID: "d7de539f-17e9-453c-af6b-e83fc7c91239"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:56.623393 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.616828 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7de539f-17e9-453c-af6b-e83fc7c91239-istio-data" pod "d7de539f-17e9-453c-af6b-e83fc7c91239" (UID: "d7de539f-17e9-453c-af6b-e83fc7c91239"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:56.639528 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.639490 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7de539f-17e9-453c-af6b-e83fc7c91239-workload-socket" pod "d7de539f-17e9-453c-af6b-e83fc7c91239" (UID: "d7de539f-17e9-453c-af6b-e83fc7c91239"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:56.851363 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.731476 4455 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7de539f-17e9-453c-af6b-e83fc7c91239-istiod-ca-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.851363 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.731514 4455 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7de539f-17e9-453c-af6b-e83fc7c91239-istio-data\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.851363 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.731530 4455 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7de539f-17e9-453c-af6b-e83fc7c91239-workload-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.851363 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.731547 4455 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7de539f-17e9-453c-af6b-e83fc7c91239-workload-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.851363 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.731562 4455 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7de539f-17e9-453c-af6b-e83fc7c91239-credential-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.851363 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.779158 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-419d32-6dff5c9996-jcbjp"] Oct 30 18:40:56.851363 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.782519 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg"] Oct 30 18:40:56.863896 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.863824 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7de539f-17e9-453c-af6b-e83fc7c91239-istio-token" pod "d7de539f-17e9-453c-af6b-e83fc7c91239" (UID: "d7de539f-17e9-453c-af6b-e83fc7c91239"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:56.947213 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.947143 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7de539f-17e9-453c-af6b-e83fc7c91239-istio-envoy" pod "d7de539f-17e9-453c-af6b-e83fc7c91239" (UID: "d7de539f-17e9-453c-af6b-e83fc7c91239"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:56.947944 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.947919 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7de539f-17e9-453c-af6b-e83fc7c91239-kube-api-access-jqbm6" pod "d7de539f-17e9-453c-af6b-e83fc7c91239" (UID: "d7de539f-17e9-453c-af6b-e83fc7c91239"). InnerVolumeSpecName "kube-api-access-jqbm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:56.950259 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.950221 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7de539f-17e9-453c-af6b-e83fc7c91239-istio-podinfo" pod "d7de539f-17e9-453c-af6b-e83fc7c91239" (UID: "d7de539f-17e9-453c-af6b-e83fc7c91239"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:40:56.962592 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.960212 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-vpvrg"] Oct 30 18:40:56.962592 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.960268 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9"] Oct 30 18:40:56.962592 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.960486 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" podUID="d2c2d808-5274-484e-8520-1f88fa79fc47" containerName="istio-proxy" containerID="cri-o://f5047262e2a0102f0504b4a2a5f81abba68df4165fafedfa87f171beafca6b7c" gracePeriod=30 Oct 30 18:40:56.973379 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.963060 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqbm6\" (UniqueName: \"kubernetes.io/projected/d7de539f-17e9-453c-af6b-e83fc7c91239-kube-api-access-jqbm6\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.973379 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.963100 4455 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7de539f-17e9-453c-af6b-e83fc7c91239-istio-podinfo\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.973379 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.963116 4455 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7de539f-17e9-453c-af6b-e83fc7c91239-istio-envoy\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.973379 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.963138 4455 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7de539f-17e9-453c-af6b-e83fc7c91239-istio-token\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.981914 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.981265 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw"] Oct 30 18:40:56.981914 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:56.981564 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" podUID="1ee94ddb-0834-4ee6-835c-4363b672e1c2" containerName="istio-proxy" containerID="cri-o://d2756a38662fd459b3cc2b877e9d404d450bcb728d8be9cd1573cb33983920d5" gracePeriod=30 Oct 30 18:40:57.026070 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:57.016607 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-istio-token" pod "d4e8e077-c9f7-44c8-9dd5-d1c7a2403862" (UID: "d4e8e077-c9f7-44c8-9dd5-d1c7a2403862"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:57.026070 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:57.025738 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-istio-podinfo" pod "d4e8e077-c9f7-44c8-9dd5-d1c7a2403862" (UID: "d4e8e077-c9f7-44c8-9dd5-d1c7a2403862"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:40:57.028534 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:57.026627 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv"] Oct 30 18:40:57.031883 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:57.030011 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" podUID="b933cb99-2e87-4707-a33b-de09b5c03536" containerName="istio-proxy" containerID="cri-o://94025eff33042a330f779ddb82ed265e6b2000e1cebc1a543c190af822e14a08" gracePeriod=30 Oct 30 18:40:57.085143 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:57.085115 4455 scope.go:122] "RemoveContainer" containerID="80af84e433f2921c01f3cb8069dc5888444f4e02f28294a6f2f627170a368f6f" Oct 30 18:40:57.110551 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:40:57.110500 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80af84e433f2921c01f3cb8069dc5888444f4e02f28294a6f2f627170a368f6f\": container with ID starting with 80af84e433f2921c01f3cb8069dc5888444f4e02f28294a6f2f627170a368f6f not found: ID does not exist" containerID="80af84e433f2921c01f3cb8069dc5888444f4e02f28294a6f2f627170a368f6f" Oct 30 18:40:57.110897 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:57.110778 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80af84e433f2921c01f3cb8069dc5888444f4e02f28294a6f2f627170a368f6f"} err="failed to get container status \"80af84e433f2921c01f3cb8069dc5888444f4e02f28294a6f2f627170a368f6f\": rpc error: code = NotFound desc = could not find container \"80af84e433f2921c01f3cb8069dc5888444f4e02f28294a6f2f627170a368f6f\": container with ID starting with 80af84e433f2921c01f3cb8069dc5888444f4e02f28294a6f2f627170a368f6f not found: ID does not exist" Oct 30 18:40:57.111032 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:57.111017 4455 scope.go:122] "RemoveContainer" containerID="938d228428904ae56b628d23ca3afa1884eb384fea6ae20f0f67618c13456691" Oct 30 18:40:57.124577 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:57.124545 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-kube-api-access-g2m4n" pod "d4e8e077-c9f7-44c8-9dd5-d1c7a2403862" (UID: "d4e8e077-c9f7-44c8-9dd5-d1c7a2403862"). InnerVolumeSpecName "kube-api-access-g2m4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:57.381887 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:57.381846 4455 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-istio-podinfo\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:57.382130 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:57.382111 4455 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-istio-token\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:57.382285 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:57.382268 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2m4n\" (UniqueName: \"kubernetes.io/projected/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862-kube-api-access-g2m4n\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:40:57.419089 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:57.419047 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" event={"ID":"d7de539f-17e9-453c-af6b-e83fc7c91239","Type":"ContainerDied","Data":"5f5ad7dbc5f2d56205d26b118b0a0506f969fe95aa0d974c1d72d7fada4b9885"} Oct 30 18:40:57.419483 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:57.419390 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584" Oct 30 18:40:57.421678 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:57.421652 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" event={"ID":"d4e8e077-c9f7-44c8-9dd5-d1c7a2403862","Type":"ContainerDied","Data":"41256cacd0db1c699eb1df7f1c3028fb3f11181a9e69117d289fbb56c6c39f99"} Oct 30 18:40:57.421916 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:57.421898 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s" Oct 30 18:40:57.446906 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:57.446884 4455 scope.go:122] "RemoveContainer" containerID="4c0afb4f15150e31f8a16b39c86d19d0b16977cddbc40c5b5cf4debe0e59c567" Oct 30 18:40:57.493257 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:57.493227 4455 scope.go:122] "RemoveContainer" containerID="909682ad7c983b900fe1e1f104fd35ebac75d470a756e2922f03fdbae688d41f" Oct 30 18:40:57.535889 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:57.535857 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s"] Oct 30 18:40:57.541110 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:57.541079 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-r4g7s"] Oct 30 18:40:57.571885 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:57.571856 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584"] Oct 30 18:40:57.580053 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:57.580017 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-29584"] Oct 30 18:40:57.639667 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:57.639548 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f13fc8d-3df1-4f51-816e-69f10c2a03cb" path="/var/lib/kubelet/pods/3f13fc8d-3df1-4f51-816e-69f10c2a03cb/volumes" Oct 30 18:40:57.640257 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:57.640233 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70d7e6c8-8d0f-40f0-a533-58158542fb7c" path="/var/lib/kubelet/pods/70d7e6c8-8d0f-40f0-a533-58158542fb7c/volumes" Oct 30 18:40:57.640719 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:57.640701 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1b171c5-abf2-4fd2-9b7f-5d103f44f704" path="/var/lib/kubelet/pods/c1b171c5-abf2-4fd2-9b7f-5d103f44f704/volumes" Oct 30 18:40:57.641222 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:57.641202 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4e8e077-c9f7-44c8-9dd5-d1c7a2403862" path="/var/lib/kubelet/pods/d4e8e077-c9f7-44c8-9dd5-d1c7a2403862/volumes" Oct 30 18:40:57.642142 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:40:57.642124 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7de539f-17e9-453c-af6b-e83fc7c91239" path="/var/lib/kubelet/pods/d7de539f-17e9-453c-af6b-e83fc7c91239/volumes" Oct 30 18:41:00.448528 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:41:00.448491 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:41:02.333131 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.333105 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" Oct 30 18:41:02.452876 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.452844 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ee94ddb-0834-4ee6-835c-4363b672e1c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ee94ddb-0834-4ee6-835c-4363b672e1c2-workload-certs\") pod \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " Oct 30 18:41:02.453226 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.452902 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ee94ddb-0834-4ee6-835c-4363b672e1c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee94ddb-0834-4ee6-835c-4363b672e1c2-credential-socket\") pod \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " Oct 30 18:41:02.453226 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.452937 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ee94ddb-0834-4ee6-835c-4363b672e1c2-kube-api-access-wb8g6\" (UniqueName: \"kubernetes.io/projected/1ee94ddb-0834-4ee6-835c-4363b672e1c2-kube-api-access-wb8g6\") pod \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " Oct 30 18:41:02.453226 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.452968 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ee94ddb-0834-4ee6-835c-4363b672e1c2-istio-token\" (UniqueName: \"kubernetes.io/projected/1ee94ddb-0834-4ee6-835c-4363b672e1c2-istio-token\") pod \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " Oct 30 18:41:02.453226 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.453002 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ee94ddb-0834-4ee6-835c-4363b672e1c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ee94ddb-0834-4ee6-835c-4363b672e1c2-istio-podinfo\") pod \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " Oct 30 18:41:02.453226 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.453107 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ee94ddb-0834-4ee6-835c-4363b672e1c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ee94ddb-0834-4ee6-835c-4363b672e1c2-istiod-ca-cert\") pod \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " Oct 30 18:41:02.453226 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.453150 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ee94ddb-0834-4ee6-835c-4363b672e1c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ee94ddb-0834-4ee6-835c-4363b672e1c2-istio-data\") pod \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " Oct 30 18:41:02.453226 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.453168 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee94ddb-0834-4ee6-835c-4363b672e1c2-workload-certs" pod "1ee94ddb-0834-4ee6-835c-4363b672e1c2" (UID: "1ee94ddb-0834-4ee6-835c-4363b672e1c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:02.453226 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.453229 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ee94ddb-0834-4ee6-835c-4363b672e1c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ee94ddb-0834-4ee6-835c-4363b672e1c2-istio-envoy\") pod \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " Oct 30 18:41:02.453848 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.453306 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ee94ddb-0834-4ee6-835c-4363b672e1c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee94ddb-0834-4ee6-835c-4363b672e1c2-workload-socket\") pod \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\" (UID: \"1ee94ddb-0834-4ee6-835c-4363b672e1c2\") " Oct 30 18:41:02.453848 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.453396 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee94ddb-0834-4ee6-835c-4363b672e1c2-credential-socket" pod "1ee94ddb-0834-4ee6-835c-4363b672e1c2" (UID: "1ee94ddb-0834-4ee6-835c-4363b672e1c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:02.453848 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.453630 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ee94ddb-0834-4ee6-835c-4363b672e1c2-istiod-ca-cert" pod "1ee94ddb-0834-4ee6-835c-4363b672e1c2" (UID: "1ee94ddb-0834-4ee6-835c-4363b672e1c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:41:02.453848 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.453737 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee94ddb-0834-4ee6-835c-4363b672e1c2-istio-data" pod "1ee94ddb-0834-4ee6-835c-4363b672e1c2" (UID: "1ee94ddb-0834-4ee6-835c-4363b672e1c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:02.454069 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.453959 4455 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ee94ddb-0834-4ee6-835c-4363b672e1c2-workload-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.454069 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.453979 4455 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee94ddb-0834-4ee6-835c-4363b672e1c2-credential-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.454069 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.453994 4455 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ee94ddb-0834-4ee6-835c-4363b672e1c2-istiod-ca-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.454069 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.454008 4455 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ee94ddb-0834-4ee6-835c-4363b672e1c2-istio-data\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.454475 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.454124 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee94ddb-0834-4ee6-835c-4363b672e1c2-workload-socket" pod "1ee94ddb-0834-4ee6-835c-4363b672e1c2" (UID: "1ee94ddb-0834-4ee6-835c-4363b672e1c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:02.455527 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.455501 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ee94ddb-0834-4ee6-835c-4363b672e1c2-istio-token" pod "1ee94ddb-0834-4ee6-835c-4363b672e1c2" (UID: "1ee94ddb-0834-4ee6-835c-4363b672e1c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:02.456147 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.456119 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ee94ddb-0834-4ee6-835c-4363b672e1c2-istio-podinfo" pod "1ee94ddb-0834-4ee6-835c-4363b672e1c2" (UID: "1ee94ddb-0834-4ee6-835c-4363b672e1c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:41:02.456262 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.456169 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee94ddb-0834-4ee6-835c-4363b672e1c2-istio-envoy" pod "1ee94ddb-0834-4ee6-835c-4363b672e1c2" (UID: "1ee94ddb-0834-4ee6-835c-4363b672e1c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:02.456262 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.456225 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ee94ddb-0834-4ee6-835c-4363b672e1c2-kube-api-access-wb8g6" pod "1ee94ddb-0834-4ee6-835c-4363b672e1c2" (UID: "1ee94ddb-0834-4ee6-835c-4363b672e1c2"). InnerVolumeSpecName "kube-api-access-wb8g6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:02.460379 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.460357 4455 generic.go:356] "Generic (PLEG): container finished" podID="d2c2d808-5274-484e-8520-1f88fa79fc47" containerID="f5047262e2a0102f0504b4a2a5f81abba68df4165fafedfa87f171beafca6b7c" exitCode=0 Oct 30 18:41:02.460465 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.460429 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" event={"ID":"d2c2d808-5274-484e-8520-1f88fa79fc47","Type":"ContainerDied","Data":"f5047262e2a0102f0504b4a2a5f81abba68df4165fafedfa87f171beafca6b7c"} Oct 30 18:41:02.461868 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.461848 4455 generic.go:356] "Generic (PLEG): container finished" podID="1ee94ddb-0834-4ee6-835c-4363b672e1c2" containerID="d2756a38662fd459b3cc2b877e9d404d450bcb728d8be9cd1573cb33983920d5" exitCode=0 Oct 30 18:41:02.461939 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.461894 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" Oct 30 18:41:02.461975 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.461929 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" event={"ID":"1ee94ddb-0834-4ee6-835c-4363b672e1c2","Type":"ContainerDied","Data":"d2756a38662fd459b3cc2b877e9d404d450bcb728d8be9cd1573cb33983920d5"} Oct 30 18:41:02.462018 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.461974 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw" event={"ID":"1ee94ddb-0834-4ee6-835c-4363b672e1c2","Type":"ContainerDied","Data":"4fecb219a18959e281b70c8eb6b506851cd427941fdb53df29dd20619ce55d35"} Oct 30 18:41:02.462018 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.462000 4455 scope.go:122] "RemoveContainer" containerID="d2756a38662fd459b3cc2b877e9d404d450bcb728d8be9cd1573cb33983920d5" Oct 30 18:41:02.463697 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.463671 4455 generic.go:356] "Generic (PLEG): container finished" podID="b933cb99-2e87-4707-a33b-de09b5c03536" containerID="94025eff33042a330f779ddb82ed265e6b2000e1cebc1a543c190af822e14a08" exitCode=0 Oct 30 18:41:02.463801 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.463733 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" event={"ID":"b933cb99-2e87-4707-a33b-de09b5c03536","Type":"ContainerDied","Data":"94025eff33042a330f779ddb82ed265e6b2000e1cebc1a543c190af822e14a08"} Oct 30 18:41:02.473360 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.473345 4455 scope.go:122] "RemoveContainer" containerID="d2756a38662fd459b3cc2b877e9d404d450bcb728d8be9cd1573cb33983920d5" Oct 30 18:41:02.473616 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:41:02.473597 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2756a38662fd459b3cc2b877e9d404d450bcb728d8be9cd1573cb33983920d5\": container with ID starting with d2756a38662fd459b3cc2b877e9d404d450bcb728d8be9cd1573cb33983920d5 not found: ID does not exist" containerID="d2756a38662fd459b3cc2b877e9d404d450bcb728d8be9cd1573cb33983920d5" Oct 30 18:41:02.473682 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.473624 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2756a38662fd459b3cc2b877e9d404d450bcb728d8be9cd1573cb33983920d5"} err="failed to get container status \"d2756a38662fd459b3cc2b877e9d404d450bcb728d8be9cd1573cb33983920d5\": rpc error: code = NotFound desc = could not find container \"d2756a38662fd459b3cc2b877e9d404d450bcb728d8be9cd1573cb33983920d5\": container with ID starting with d2756a38662fd459b3cc2b877e9d404d450bcb728d8be9cd1573cb33983920d5 not found: ID does not exist" Oct 30 18:41:02.512812 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.512781 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw"] Oct 30 18:41:02.518842 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.518816 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-wdhjw"] Oct 30 18:41:02.554959 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.554930 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb8g6\" (UniqueName: \"kubernetes.io/projected/1ee94ddb-0834-4ee6-835c-4363b672e1c2-kube-api-access-wb8g6\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.554959 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.554961 4455 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ee94ddb-0834-4ee6-835c-4363b672e1c2-istio-token\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.555162 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.554972 4455 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ee94ddb-0834-4ee6-835c-4363b672e1c2-istio-podinfo\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.555162 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.554980 4455 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ee94ddb-0834-4ee6-835c-4363b672e1c2-istio-envoy\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.555162 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.554989 4455 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee94ddb-0834-4ee6-835c-4363b672e1c2-workload-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.690167 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.690135 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" Oct 30 18:41:02.864765 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.864665 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2c2d808-5274-484e-8520-1f88fa79fc47-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c2d808-5274-484e-8520-1f88fa79fc47-istio-data\") pod \"d2c2d808-5274-484e-8520-1f88fa79fc47\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " Oct 30 18:41:02.864765 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.864747 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2c2d808-5274-484e-8520-1f88fa79fc47-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c2d808-5274-484e-8520-1f88fa79fc47-istio-podinfo\") pod \"d2c2d808-5274-484e-8520-1f88fa79fc47\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " Oct 30 18:41:02.864965 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.864788 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2c2d808-5274-484e-8520-1f88fa79fc47-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c2d808-5274-484e-8520-1f88fa79fc47-istiod-ca-cert\") pod \"d2c2d808-5274-484e-8520-1f88fa79fc47\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " Oct 30 18:41:02.864965 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.864822 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2c2d808-5274-484e-8520-1f88fa79fc47-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c2d808-5274-484e-8520-1f88fa79fc47-workload-certs\") pod \"d2c2d808-5274-484e-8520-1f88fa79fc47\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " Oct 30 18:41:02.864965 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.864847 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2c2d808-5274-484e-8520-1f88fa79fc47-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c2d808-5274-484e-8520-1f88fa79fc47-credential-socket\") pod \"d2c2d808-5274-484e-8520-1f88fa79fc47\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " Oct 30 18:41:02.864965 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.864891 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2c2d808-5274-484e-8520-1f88fa79fc47-kube-api-access-vwnmj\" (UniqueName: \"kubernetes.io/projected/d2c2d808-5274-484e-8520-1f88fa79fc47-kube-api-access-vwnmj\") pod \"d2c2d808-5274-484e-8520-1f88fa79fc47\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " Oct 30 18:41:02.864965 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.864955 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2c2d808-5274-484e-8520-1f88fa79fc47-istio-token\" (UniqueName: \"kubernetes.io/projected/d2c2d808-5274-484e-8520-1f88fa79fc47-istio-token\") pod \"d2c2d808-5274-484e-8520-1f88fa79fc47\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " Oct 30 18:41:02.865230 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.864988 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2c2d808-5274-484e-8520-1f88fa79fc47-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c2d808-5274-484e-8520-1f88fa79fc47-workload-socket\") pod \"d2c2d808-5274-484e-8520-1f88fa79fc47\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " Oct 30 18:41:02.865230 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.864986 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c2d808-5274-484e-8520-1f88fa79fc47-istio-data" pod "d2c2d808-5274-484e-8520-1f88fa79fc47" (UID: "d2c2d808-5274-484e-8520-1f88fa79fc47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:02.865230 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.865041 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2c2d808-5274-484e-8520-1f88fa79fc47-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c2d808-5274-484e-8520-1f88fa79fc47-istio-envoy\") pod \"d2c2d808-5274-484e-8520-1f88fa79fc47\" (UID: \"d2c2d808-5274-484e-8520-1f88fa79fc47\") " Oct 30 18:41:02.865230 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.865142 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c2d808-5274-484e-8520-1f88fa79fc47-workload-certs" pod "d2c2d808-5274-484e-8520-1f88fa79fc47" (UID: "d2c2d808-5274-484e-8520-1f88fa79fc47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:02.865496 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.865459 4455 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c2d808-5274-484e-8520-1f88fa79fc47-istio-data\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.865496 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.865485 4455 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c2d808-5274-484e-8520-1f88fa79fc47-workload-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.865913 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.865875 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c2d808-5274-484e-8520-1f88fa79fc47-credential-socket" pod "d2c2d808-5274-484e-8520-1f88fa79fc47" (UID: "d2c2d808-5274-484e-8520-1f88fa79fc47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:02.866249 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.866215 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c2d808-5274-484e-8520-1f88fa79fc47-workload-socket" pod "d2c2d808-5274-484e-8520-1f88fa79fc47" (UID: "d2c2d808-5274-484e-8520-1f88fa79fc47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:02.866462 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.866439 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2c2d808-5274-484e-8520-1f88fa79fc47-istiod-ca-cert" pod "d2c2d808-5274-484e-8520-1f88fa79fc47" (UID: "d2c2d808-5274-484e-8520-1f88fa79fc47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:41:02.867775 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.867754 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c2d808-5274-484e-8520-1f88fa79fc47-istio-envoy" pod "d2c2d808-5274-484e-8520-1f88fa79fc47" (UID: "d2c2d808-5274-484e-8520-1f88fa79fc47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:02.867892 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.867857 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2c2d808-5274-484e-8520-1f88fa79fc47-istio-podinfo" pod "d2c2d808-5274-484e-8520-1f88fa79fc47" (UID: "d2c2d808-5274-484e-8520-1f88fa79fc47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:41:02.868085 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.868068 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2c2d808-5274-484e-8520-1f88fa79fc47-kube-api-access-vwnmj" pod "d2c2d808-5274-484e-8520-1f88fa79fc47" (UID: "d2c2d808-5274-484e-8520-1f88fa79fc47"). InnerVolumeSpecName "kube-api-access-vwnmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:02.868351 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.868316 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2c2d808-5274-484e-8520-1f88fa79fc47-istio-token" pod "d2c2d808-5274-484e-8520-1f88fa79fc47" (UID: "d2c2d808-5274-484e-8520-1f88fa79fc47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:02.967165 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.967097 4455 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c2d808-5274-484e-8520-1f88fa79fc47-istio-podinfo\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.967165 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.967148 4455 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c2d808-5274-484e-8520-1f88fa79fc47-istiod-ca-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.967165 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.967164 4455 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c2d808-5274-484e-8520-1f88fa79fc47-credential-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.967679 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.967178 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwnmj\" (UniqueName: \"kubernetes.io/projected/d2c2d808-5274-484e-8520-1f88fa79fc47-kube-api-access-vwnmj\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.967679 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.967194 4455 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c2d808-5274-484e-8520-1f88fa79fc47-istio-token\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.967679 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.967206 4455 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c2d808-5274-484e-8520-1f88fa79fc47-workload-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.967679 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:02.967220 4455 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c2d808-5274-484e-8520-1f88fa79fc47-istio-envoy\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:41:03.003283 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.003258 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" Oct 30 18:41:03.169505 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.169408 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b933cb99-2e87-4707-a33b-de09b5c03536-istio-token\" (UniqueName: \"kubernetes.io/projected/b933cb99-2e87-4707-a33b-de09b5c03536-istio-token\") pod \"b933cb99-2e87-4707-a33b-de09b5c03536\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " Oct 30 18:41:03.169505 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.169466 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b933cb99-2e87-4707-a33b-de09b5c03536-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b933cb99-2e87-4707-a33b-de09b5c03536-workload-certs\") pod \"b933cb99-2e87-4707-a33b-de09b5c03536\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " Oct 30 18:41:03.169505 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.169509 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b933cb99-2e87-4707-a33b-de09b5c03536-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b933cb99-2e87-4707-a33b-de09b5c03536-credential-socket\") pod \"b933cb99-2e87-4707-a33b-de09b5c03536\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " Oct 30 18:41:03.169804 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.169553 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b933cb99-2e87-4707-a33b-de09b5c03536-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b933cb99-2e87-4707-a33b-de09b5c03536-istio-podinfo\") pod \"b933cb99-2e87-4707-a33b-de09b5c03536\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " Oct 30 18:41:03.169804 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.169585 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b933cb99-2e87-4707-a33b-de09b5c03536-kube-api-access-rnfvq\" (UniqueName: \"kubernetes.io/projected/b933cb99-2e87-4707-a33b-de09b5c03536-kube-api-access-rnfvq\") pod \"b933cb99-2e87-4707-a33b-de09b5c03536\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " Oct 30 18:41:03.169804 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.169619 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b933cb99-2e87-4707-a33b-de09b5c03536-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b933cb99-2e87-4707-a33b-de09b5c03536-workload-socket\") pod \"b933cb99-2e87-4707-a33b-de09b5c03536\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " Oct 30 18:41:03.169804 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.169646 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b933cb99-2e87-4707-a33b-de09b5c03536-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b933cb99-2e87-4707-a33b-de09b5c03536-istio-envoy\") pod \"b933cb99-2e87-4707-a33b-de09b5c03536\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " Oct 30 18:41:03.169804 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.169745 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b933cb99-2e87-4707-a33b-de09b5c03536-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b933cb99-2e87-4707-a33b-de09b5c03536-istiod-ca-cert\") pod \"b933cb99-2e87-4707-a33b-de09b5c03536\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " Oct 30 18:41:03.169804 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.169773 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b933cb99-2e87-4707-a33b-de09b5c03536-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b933cb99-2e87-4707-a33b-de09b5c03536-istio-data\") pod \"b933cb99-2e87-4707-a33b-de09b5c03536\" (UID: \"b933cb99-2e87-4707-a33b-de09b5c03536\") " Oct 30 18:41:03.170118 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.169823 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b933cb99-2e87-4707-a33b-de09b5c03536-credential-socket" pod "b933cb99-2e87-4707-a33b-de09b5c03536" (UID: "b933cb99-2e87-4707-a33b-de09b5c03536"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:03.170118 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.169845 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b933cb99-2e87-4707-a33b-de09b5c03536-workload-certs" pod "b933cb99-2e87-4707-a33b-de09b5c03536" (UID: "b933cb99-2e87-4707-a33b-de09b5c03536"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:03.170118 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.170109 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b933cb99-2e87-4707-a33b-de09b5c03536-istiod-ca-cert" pod "b933cb99-2e87-4707-a33b-de09b5c03536" (UID: "b933cb99-2e87-4707-a33b-de09b5c03536"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:41:03.170501 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.170118 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b933cb99-2e87-4707-a33b-de09b5c03536-istio-data" pod "b933cb99-2e87-4707-a33b-de09b5c03536" (UID: "b933cb99-2e87-4707-a33b-de09b5c03536"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:03.170501 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.170252 4455 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b933cb99-2e87-4707-a33b-de09b5c03536-istiod-ca-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:41:03.170501 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.170273 4455 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b933cb99-2e87-4707-a33b-de09b5c03536-istio-data\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:41:03.170501 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.170276 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b933cb99-2e87-4707-a33b-de09b5c03536-workload-socket" pod "b933cb99-2e87-4707-a33b-de09b5c03536" (UID: "b933cb99-2e87-4707-a33b-de09b5c03536"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:03.170501 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.170289 4455 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b933cb99-2e87-4707-a33b-de09b5c03536-workload-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:41:03.170501 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.170302 4455 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b933cb99-2e87-4707-a33b-de09b5c03536-credential-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:41:03.172008 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.171983 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b933cb99-2e87-4707-a33b-de09b5c03536-istio-token" pod "b933cb99-2e87-4707-a33b-de09b5c03536" (UID: "b933cb99-2e87-4707-a33b-de09b5c03536"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:03.172122 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.172054 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b933cb99-2e87-4707-a33b-de09b5c03536-kube-api-access-rnfvq" pod "b933cb99-2e87-4707-a33b-de09b5c03536" (UID: "b933cb99-2e87-4707-a33b-de09b5c03536"). InnerVolumeSpecName "kube-api-access-rnfvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:03.172607 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.172588 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b933cb99-2e87-4707-a33b-de09b5c03536-istio-envoy" pod "b933cb99-2e87-4707-a33b-de09b5c03536" (UID: "b933cb99-2e87-4707-a33b-de09b5c03536"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:03.172689 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.172603 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b933cb99-2e87-4707-a33b-de09b5c03536-istio-podinfo" pod "b933cb99-2e87-4707-a33b-de09b5c03536" (UID: "b933cb99-2e87-4707-a33b-de09b5c03536"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:41:03.271179 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.271145 4455 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b933cb99-2e87-4707-a33b-de09b5c03536-istio-token\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:41:03.271179 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.271175 4455 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b933cb99-2e87-4707-a33b-de09b5c03536-istio-podinfo\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:41:03.271179 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.271185 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnfvq\" (UniqueName: \"kubernetes.io/projected/b933cb99-2e87-4707-a33b-de09b5c03536-kube-api-access-rnfvq\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:41:03.271451 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.271195 4455 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b933cb99-2e87-4707-a33b-de09b5c03536-workload-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:41:03.271451 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.271206 4455 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b933cb99-2e87-4707-a33b-de09b5c03536-istio-envoy\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:41:03.472928 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.472897 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" Oct 30 18:41:03.472928 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.472920 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9" event={"ID":"d2c2d808-5274-484e-8520-1f88fa79fc47","Type":"ContainerDied","Data":"ab4e174b785d423cfa8c20dff3c45370d181a767140eeee606e826f9570cda77"} Oct 30 18:41:03.473577 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.472988 4455 scope.go:122] "RemoveContainer" containerID="f5047262e2a0102f0504b4a2a5f81abba68df4165fafedfa87f171beafca6b7c" Oct 30 18:41:03.477975 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.477940 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" Oct 30 18:41:03.478120 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.477981 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv" event={"ID":"b933cb99-2e87-4707-a33b-de09b5c03536","Type":"ContainerDied","Data":"125fa24784c534b6f3dbcb38ea37748d945cbb0bfacea8203f2912e79eaffb61"} Oct 30 18:41:03.486146 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.485988 4455 scope.go:122] "RemoveContainer" containerID="94025eff33042a330f779ddb82ed265e6b2000e1cebc1a543c190af822e14a08" Oct 30 18:41:03.524377 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.524345 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9"] Oct 30 18:41:03.527830 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.527804 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-lctr9"] Oct 30 18:41:03.562780 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.562755 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv"] Oct 30 18:41:03.570008 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.569980 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-m4txv"] Oct 30 18:41:03.635847 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.635814 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ee94ddb-0834-4ee6-835c-4363b672e1c2" path="/var/lib/kubelet/pods/1ee94ddb-0834-4ee6-835c-4363b672e1c2/volumes" Oct 30 18:41:03.636333 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.636304 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b933cb99-2e87-4707-a33b-de09b5c03536" path="/var/lib/kubelet/pods/b933cb99-2e87-4707-a33b-de09b5c03536/volumes" Oct 30 18:41:03.636735 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:03.636721 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2c2d808-5274-484e-8520-1f88fa79fc47" path="/var/lib/kubelet/pods/d2c2d808-5274-484e-8520-1f88fa79fc47/volumes" Oct 30 18:41:14.244825 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.244748 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs"] Oct 30 18:41:14.245366 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245349 4455 cpu_manager.go:414] "containerMap: removing container" podUID="70d7e6c8-8d0f-40f0-a533-58158542fb7c" containerName="istio-proxy" Oct 30 18:41:14.245427 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245369 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70d7e6c8-8d0f-40f0-a533-58158542fb7c" containerName="istio-proxy" Oct 30 18:41:14.245427 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245387 4455 cpu_manager.go:414] "containerMap: removing container" podUID="4936cb40-0694-428e-bfd1-02a96a72a8a8" containerName="maas-api" Oct 30 18:41:14.245427 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245393 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4936cb40-0694-428e-bfd1-02a96a72a8a8" containerName="maas-api" Oct 30 18:41:14.245427 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245404 4455 cpu_manager.go:414] "containerMap: removing container" podUID="3f13fc8d-3df1-4f51-816e-69f10c2a03cb" containerName="payload-pre-processing" Oct 30 18:41:14.245427 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245410 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f13fc8d-3df1-4f51-816e-69f10c2a03cb" containerName="payload-pre-processing" Oct 30 18:41:14.245427 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245417 4455 cpu_manager.go:414] "containerMap: removing container" podUID="2b13dbfe-f5db-46e6-810e-9d4ac231c94c" containerName="payload-pre-processing" Oct 30 18:41:14.245427 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245422 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b13dbfe-f5db-46e6-810e-9d4ac231c94c" containerName="payload-pre-processing" Oct 30 18:41:14.245656 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245438 4455 cpu_manager.go:414] "containerMap: removing container" podUID="d2c2d808-5274-484e-8520-1f88fa79fc47" containerName="istio-proxy" Oct 30 18:41:14.245656 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245487 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2c2d808-5274-484e-8520-1f88fa79fc47" containerName="istio-proxy" Oct 30 18:41:14.245656 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245499 4455 cpu_manager.go:414] "containerMap: removing container" podUID="a11771b6-cb92-4783-bd70-4eb964d94f00" containerName="maas-api" Oct 30 18:41:14.245656 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245506 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a11771b6-cb92-4783-bd70-4eb964d94f00" containerName="maas-api" Oct 30 18:41:14.245656 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245518 4455 cpu_manager.go:414] "containerMap: removing container" podUID="ac80ccea-a611-4de1-951c-479ad3d827a7" containerName="istio-proxy" Oct 30 18:41:14.245656 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245525 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac80ccea-a611-4de1-951c-479ad3d827a7" containerName="istio-proxy" Oct 30 18:41:14.245656 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245536 4455 cpu_manager.go:414] "containerMap: removing container" podUID="67a8b3a3-4d72-4077-875f-a0262982c7e9" containerName="payload-pre-processing" Oct 30 18:41:14.245656 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245543 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67a8b3a3-4d72-4077-875f-a0262982c7e9" containerName="payload-pre-processing" Oct 30 18:41:14.245656 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245557 4455 cpu_manager.go:414] "containerMap: removing container" podUID="e62b028a-4ecb-46f8-9d6c-94bf236dc782" containerName="maas-api" Oct 30 18:41:14.245656 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245564 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e62b028a-4ecb-46f8-9d6c-94bf236dc782" containerName="maas-api" Oct 30 18:41:14.245656 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245576 4455 cpu_manager.go:414] "containerMap: removing container" podUID="b41517e2-cc07-4530-8ec9-092c2a92ad47" containerName="istio-proxy" Oct 30 18:41:14.245656 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245583 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b41517e2-cc07-4530-8ec9-092c2a92ad47" containerName="istio-proxy" Oct 30 18:41:14.245656 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245590 4455 cpu_manager.go:414] "containerMap: removing container" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" Oct 30 18:41:14.245656 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245594 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" Oct 30 18:41:14.245656 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245602 4455 cpu_manager.go:414] "containerMap: removing container" podUID="b933cb99-2e87-4707-a33b-de09b5c03536" containerName="istio-proxy" Oct 30 18:41:14.245656 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245607 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b933cb99-2e87-4707-a33b-de09b5c03536" containerName="istio-proxy" Oct 30 18:41:14.245656 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245612 4455 cpu_manager.go:414] "containerMap: removing container" podUID="13f59414-113d-4863-88b8-e43c25b17cff" containerName="payload-pre-processing" Oct 30 18:41:14.245656 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245618 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13f59414-113d-4863-88b8-e43c25b17cff" containerName="payload-pre-processing" Oct 30 18:41:14.245656 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245627 4455 cpu_manager.go:414] "containerMap: removing container" podUID="03cfd14d-26f3-4294-a58d-9e47ad8b1b67" containerName="payload-processing" Oct 30 18:41:14.245656 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245631 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03cfd14d-26f3-4294-a58d-9e47ad8b1b67" containerName="payload-processing" Oct 30 18:41:14.245656 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245642 4455 cpu_manager.go:414] "containerMap: removing container" podUID="4deaf1e2-755d-41bb-b3b7-ff08a88186d0" containerName="maas-api" Oct 30 18:41:14.245656 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245646 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4deaf1e2-755d-41bb-b3b7-ff08a88186d0" containerName="maas-api" Oct 30 18:41:14.245656 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245654 4455 cpu_manager.go:414] "containerMap: removing container" podUID="d6058153-6f0e-4d2f-ac5c-53be26e376be" containerName="maas-api" Oct 30 18:41:14.245656 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245658 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6058153-6f0e-4d2f-ac5c-53be26e376be" containerName="maas-api" Oct 30 18:41:14.245656 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245665 4455 cpu_manager.go:414] "containerMap: removing container" podUID="baac9ab5-2caa-4596-a802-46062f6fcb4c" containerName="payload-pre-processing" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245669 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baac9ab5-2caa-4596-a802-46062f6fcb4c" containerName="payload-pre-processing" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245677 4455 cpu_manager.go:414] "containerMap: removing container" podUID="7c9b7598-cec3-43ab-90cf-93e7b407d68e" containerName="istio-proxy" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245681 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c9b7598-cec3-43ab-90cf-93e7b407d68e" containerName="istio-proxy" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245700 4455 cpu_manager.go:414] "containerMap: removing container" podUID="038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5" containerName="maas-api" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245706 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5" containerName="maas-api" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245713 4455 cpu_manager.go:414] "containerMap: removing container" podUID="bbf4bc50-3670-40de-8159-3d8805fdd601" containerName="istio-proxy" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245717 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbf4bc50-3670-40de-8159-3d8805fdd601" containerName="istio-proxy" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245726 4455 cpu_manager.go:414] "containerMap: removing container" podUID="93b710b6-b5a8-4fe0-ac70-861dd036b70d" containerName="istio-proxy" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245730 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93b710b6-b5a8-4fe0-ac70-861dd036b70d" containerName="istio-proxy" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245739 4455 cpu_manager.go:414] "containerMap: removing container" podUID="b32c8229-35fa-4eff-abf2-4d3b8713b1eb" containerName="payload-pre-processing" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245743 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b32c8229-35fa-4eff-abf2-4d3b8713b1eb" containerName="payload-pre-processing" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245749 4455 cpu_manager.go:414] "containerMap: removing container" podUID="67ec911d-b41b-49f5-b1e3-4637bc7c48f3" containerName="payload-processing" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245753 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67ec911d-b41b-49f5-b1e3-4637bc7c48f3" containerName="payload-processing" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245764 4455 cpu_manager.go:414] "containerMap: removing container" podUID="a8a6e230-4b8f-4d8d-af42-d6660522152e" containerName="istio-proxy" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245768 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8a6e230-4b8f-4d8d-af42-d6660522152e" containerName="istio-proxy" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245778 4455 cpu_manager.go:414] "containerMap: removing container" podUID="d4e8e077-c9f7-44c8-9dd5-d1c7a2403862" containerName="istio-proxy" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245782 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4e8e077-c9f7-44c8-9dd5-d1c7a2403862" containerName="istio-proxy" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245788 4455 cpu_manager.go:414] "containerMap: removing container" podUID="3daa8589-9def-486f-82ee-0c923a18b68b" containerName="istio-proxy" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245792 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3daa8589-9def-486f-82ee-0c923a18b68b" containerName="istio-proxy" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245798 4455 cpu_manager.go:414] "containerMap: removing container" podUID="d7de539f-17e9-453c-af6b-e83fc7c91239" containerName="istio-proxy" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245804 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7de539f-17e9-453c-af6b-e83fc7c91239" containerName="istio-proxy" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245813 4455 cpu_manager.go:414] "containerMap: removing container" podUID="1b032444-b7c0-4d3c-8a07-aee2df84d8bd" containerName="payload-processing" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245817 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b032444-b7c0-4d3c-8a07-aee2df84d8bd" containerName="payload-processing" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245827 4455 cpu_manager.go:414] "containerMap: removing container" podUID="1ee94ddb-0834-4ee6-835c-4363b672e1c2" containerName="istio-proxy" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245831 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ee94ddb-0834-4ee6-835c-4363b672e1c2" containerName="istio-proxy" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245838 4455 cpu_manager.go:414] "containerMap: removing container" podUID="c63830ed-816d-4e24-9900-7ee260bf622f" containerName="payload-processing" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245843 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c63830ed-816d-4e24-9900-7ee260bf622f" containerName="payload-processing" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245848 4455 cpu_manager.go:414] "containerMap: removing container" podUID="c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff" containerName="payload-processing" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245852 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff" containerName="payload-processing" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245861 4455 cpu_manager.go:414] "containerMap: removing container" podUID="fe742589-45cc-4159-b8ba-b4402856f2db" containerName="maas-api" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245865 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe742589-45cc-4159-b8ba-b4402856f2db" containerName="maas-api" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245871 4455 cpu_manager.go:414] "containerMap: removing container" podUID="502021d0-cc42-404e-a0f7-67f7392c82bf" containerName="payload-processing" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245876 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="502021d0-cc42-404e-a0f7-67f7392c82bf" containerName="payload-processing" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245884 4455 cpu_manager.go:414] "containerMap: removing container" podUID="099a77ed-9f6a-4937-85c1-b7053bc602fa" containerName="istio-proxy" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245888 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="099a77ed-9f6a-4937-85c1-b7053bc602fa" containerName="istio-proxy" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245898 4455 cpu_manager.go:414] "containerMap: removing container" podUID="1ae8cd58-07cf-40ae-9421-d2c91aee0379" containerName="payload-pre-processing" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245903 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ae8cd58-07cf-40ae-9421-d2c91aee0379" containerName="payload-pre-processing" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245910 4455 cpu_manager.go:414] "containerMap: removing container" podUID="c1b171c5-abf2-4fd2-9b7f-5d103f44f704" containerName="payload-processing" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245914 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1b171c5-abf2-4fd2-9b7f-5d103f44f704" containerName="payload-processing" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245920 4455 cpu_manager.go:414] "containerMap: removing container" podUID="e972725e-489d-4f2a-a734-27d7eb570858" containerName="istio-proxy" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.245924 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e972725e-489d-4f2a-a734-27d7eb570858" containerName="istio-proxy" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246034 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbf4bc50-3670-40de-8159-3d8805fdd601" containerName="istio-proxy" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246042 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="93b710b6-b5a8-4fe0-ac70-861dd036b70d" containerName="istio-proxy" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246050 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="b32c8229-35fa-4eff-abf2-4d3b8713b1eb" containerName="payload-pre-processing" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246056 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="67ec911d-b41b-49f5-b1e3-4637bc7c48f3" containerName="payload-processing" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246064 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8a6e230-4b8f-4d8d-af42-d6660522152e" containerName="istio-proxy" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246070 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4e8e077-c9f7-44c8-9dd5-d1c7a2403862" containerName="istio-proxy" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246075 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="3daa8589-9def-486f-82ee-0c923a18b68b" containerName="istio-proxy" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246081 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7de539f-17e9-453c-af6b-e83fc7c91239" containerName="istio-proxy" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246085 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b032444-b7c0-4d3c-8a07-aee2df84d8bd" containerName="payload-processing" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246089 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ee94ddb-0834-4ee6-835c-4363b672e1c2" containerName="istio-proxy" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246094 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="c63830ed-816d-4e24-9900-7ee260bf622f" containerName="payload-processing" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246099 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4d0abb6-a0b5-4c39-88e0-a00cc1bba6ff" containerName="payload-processing" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246105 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe742589-45cc-4159-b8ba-b4402856f2db" containerName="maas-api" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246109 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="502021d0-cc42-404e-a0f7-67f7392c82bf" containerName="payload-processing" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246114 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="099a77ed-9f6a-4937-85c1-b7053bc602fa" containerName="istio-proxy" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246118 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="e62b028a-4ecb-46f8-9d6c-94bf236dc782" containerName="maas-api" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246123 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ae8cd58-07cf-40ae-9421-d2c91aee0379" containerName="payload-pre-processing" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246128 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="e972725e-489d-4f2a-a734-27d7eb570858" containerName="istio-proxy" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246134 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1b171c5-abf2-4fd2-9b7f-5d103f44f704" containerName="payload-processing" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246140 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="70d7e6c8-8d0f-40f0-a533-58158542fb7c" containerName="istio-proxy" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246145 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="4936cb40-0694-428e-bfd1-02a96a72a8a8" containerName="maas-api" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246150 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f13fc8d-3df1-4f51-816e-69f10c2a03cb" containerName="payload-pre-processing" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246155 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b13dbfe-f5db-46e6-810e-9d4ac231c94c" containerName="payload-pre-processing" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246159 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2c2d808-5274-484e-8520-1f88fa79fc47" containerName="istio-proxy" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246166 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="a11771b6-cb92-4783-bd70-4eb964d94f00" containerName="maas-api" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246170 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac80ccea-a611-4de1-951c-479ad3d827a7" containerName="istio-proxy" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246176 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="67a8b3a3-4d72-4077-875f-a0262982c7e9" containerName="payload-pre-processing" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246181 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="e62b028a-4ecb-46f8-9d6c-94bf236dc782" containerName="maas-api" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246186 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="b41517e2-cc07-4530-8ec9-092c2a92ad47" containerName="istio-proxy" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246191 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3ad09e5-ba36-401c-b93b-32bcd2f6c5c7" containerName="main" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246196 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="b933cb99-2e87-4707-a33b-de09b5c03536" containerName="istio-proxy" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246201 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="13f59414-113d-4863-88b8-e43c25b17cff" containerName="payload-pre-processing" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246207 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="03cfd14d-26f3-4294-a58d-9e47ad8b1b67" containerName="payload-processing" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246211 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="4deaf1e2-755d-41bb-b3b7-ff08a88186d0" containerName="maas-api" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246216 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6058153-6f0e-4d2f-ac5c-53be26e376be" containerName="maas-api" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246221 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="baac9ab5-2caa-4596-a802-46062f6fcb4c" containerName="payload-pre-processing" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246228 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c9b7598-cec3-43ab-90cf-93e7b407d68e" containerName="istio-proxy" Oct 30 18:41:14.246438 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.246233 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="038dfdf8-c4a1-4ba3-91bf-b7ee685eb5f5" containerName="maas-api" Oct 30 18:41:14.252162 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.252143 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" Oct 30 18:41:14.266714 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.266680 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-ac422f-openshift-default-dockercfg-w59vs\"" Oct 30 18:41:14.283039 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.283002 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs"] Oct 30 18:41:14.325783 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.325750 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-istio-token\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" Oct 30 18:41:14.325998 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.325804 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-credential-socket\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" Oct 30 18:41:14.325998 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.325829 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-workload-certs\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" Oct 30 18:41:14.326109 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.325984 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-workload-socket\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" Oct 30 18:41:14.326109 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.326088 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-istio-data\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" Oct 30 18:41:14.326182 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.326145 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvbn7\" (UniqueName: \"kubernetes.io/projected/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-kube-api-access-bvbn7\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" Oct 30 18:41:14.326237 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.326183 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-istio-envoy\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" Oct 30 18:41:14.326330 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.326297 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-istio-podinfo\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" Oct 30 18:41:14.326373 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.326361 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-istiod-ca-cert\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" Oct 30 18:41:14.427214 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.427174 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-istio-data\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" Oct 30 18:41:14.427434 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.427247 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvbn7\" (UniqueName: \"kubernetes.io/projected/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-kube-api-access-bvbn7\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" Oct 30 18:41:14.427434 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.427294 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-istio-envoy\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" Oct 30 18:41:14.427434 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.427373 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-istio-podinfo\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" Oct 30 18:41:14.427434 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.427427 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-istiod-ca-cert\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" Oct 30 18:41:14.427691 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.427466 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-istio-token\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" Oct 30 18:41:14.427691 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.427519 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-credential-socket\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" Oct 30 18:41:14.427691 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.427555 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-workload-certs\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" Oct 30 18:41:14.427691 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.427620 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-workload-socket\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" Oct 30 18:41:14.427691 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.427631 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-istio-data\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" Oct 30 18:41:14.427971 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.427879 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-credential-socket\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" Oct 30 18:41:14.428028 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.427973 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-workload-certs\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" Oct 30 18:41:14.428028 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.427985 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-workload-socket\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" Oct 30 18:41:14.428180 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.428160 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-istiod-ca-cert\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" Oct 30 18:41:14.430001 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.429968 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-istio-envoy\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" Oct 30 18:41:14.430119 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.430064 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-istio-podinfo\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" Oct 30 18:41:14.452830 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.452805 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-istio-token\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" Oct 30 18:41:14.453044 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.453020 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvbn7\" (UniqueName: \"kubernetes.io/projected/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-kube-api-access-bvbn7\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" Oct 30 18:41:14.567232 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.567126 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" Oct 30 18:41:14.901579 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.901495 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs"] Oct 30 18:41:14.904953 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:41:14.904922 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15651f51_8abe_47fa_89f8_a9b30ee0b3e9.slice/crio-713a6cc982202eeb9faa44cbd43be6f1a3d0d1c274fadaf8d4df4cef2115b237 WatchSource:0}: Error finding container 713a6cc982202eeb9faa44cbd43be6f1a3d0d1c274fadaf8d4df4cef2115b237: Status 404 returned error can't find the container with id 713a6cc982202eeb9faa44cbd43be6f1a3d0d1c274fadaf8d4df4cef2115b237 Oct 30 18:41:14.907562 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.907528 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:41:14.907643 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.907605 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:41:14.907643 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:14.907635 4455 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 18:41:15.607537 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:15.607492 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" event={"ID":"15651f51-8abe-47fa-89f8-a9b30ee0b3e9","Type":"ContainerStarted","Data":"d84286ac03dcb40d0c6ce75d2fa09e0bcde97fac943fb949f4924fdc17ce39ac"} Oct 30 18:41:15.608079 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:15.607580 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" event={"ID":"15651f51-8abe-47fa-89f8-a9b30ee0b3e9","Type":"ContainerStarted","Data":"713a6cc982202eeb9faa44cbd43be6f1a3d0d1c274fadaf8d4df4cef2115b237"} Oct 30 18:41:15.687349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:15.677715 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" podStartSLOduration=1.677693028 podStartE2EDuration="1.677693028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:15.663625175 +0000 UTC m=+2928.151262210" watchObservedRunningTime="2026-10-30 18:41:15.677693028 +0000 UTC m=+2928.165330062" Oct 30 18:41:16.571922 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:16.571888 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" Oct 30 18:41:16.572279 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:16.572253 4455 patch_prober.go:29] interesting pod/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Oct 30 18:41:16.572507 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:16.572474 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" podUID="15651f51-8abe-47fa-89f8-a9b30ee0b3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Oct 30 18:41:17.569711 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:17.569671 4455 patch_prober.go:29] interesting pod/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Oct 30 18:41:17.570308 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:17.570271 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" podUID="15651f51-8abe-47fa-89f8-a9b30ee0b3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Oct 30 18:41:18.568777 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:18.568742 4455 patch_prober.go:29] interesting pod/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Oct 30 18:41:18.569105 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:18.569065 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" podUID="15651f51-8abe-47fa-89f8-a9b30ee0b3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Oct 30 18:41:19.575347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:19.571482 4455 patch_prober.go:29] interesting pod/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Oct 30 18:41:19.575347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:19.571538 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" podUID="15651f51-8abe-47fa-89f8-a9b30ee0b3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Oct 30 18:41:20.585898 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:20.585858 4455 patch_prober.go:29] interesting pod/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Oct 30 18:41:20.586564 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:20.586529 4455 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" podUID="15651f51-8abe-47fa-89f8-a9b30ee0b3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Oct 30 18:41:21.635879 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:21.635847 4455 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" Oct 30 18:41:21.636605 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:21.636579 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" Oct 30 18:41:21.637659 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:21.637631 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" Oct 30 18:41:30.388309 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:41:30.388262 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:41:33.704052 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:33.704019 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4"] Oct 30 18:41:33.704634 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:33.704245 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4" podUID="9d0733d2-85ad-4966-9740-97fffe7f54f6" containerName="maas-api" containerID="cri-o://8ed2643be90cff51f72e2ea02dfba6ff86cad80c591e675d79875db9b5f42f1e" gracePeriod=30 Oct 30 18:41:34.973564 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:34.973538 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4" Oct 30 18:41:35.104641 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:35.104554 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d0733d2-85ad-4966-9740-97fffe7f54f6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d0733d2-85ad-4966-9740-97fffe7f54f6-metrics-certs\") pod \"9d0733d2-85ad-4966-9740-97fffe7f54f6\" (UID: \"9d0733d2-85ad-4966-9740-97fffe7f54f6\") " Oct 30 18:41:35.104641 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:35.104635 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d0733d2-85ad-4966-9740-97fffe7f54f6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d0733d2-85ad-4966-9740-97fffe7f54f6-maas-api-tls\") pod \"9d0733d2-85ad-4966-9740-97fffe7f54f6\" (UID: \"9d0733d2-85ad-4966-9740-97fffe7f54f6\") " Oct 30 18:41:35.104895 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:35.104665 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d0733d2-85ad-4966-9740-97fffe7f54f6-kube-api-access-5qq2q\" (UniqueName: \"kubernetes.io/projected/9d0733d2-85ad-4966-9740-97fffe7f54f6-kube-api-access-5qq2q\") pod \"9d0733d2-85ad-4966-9740-97fffe7f54f6\" (UID: \"9d0733d2-85ad-4966-9740-97fffe7f54f6\") " Oct 30 18:41:35.104895 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:35.104784 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d0733d2-85ad-4966-9740-97fffe7f54f6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d0733d2-85ad-4966-9740-97fffe7f54f6-openshift-service-ca\") pod \"9d0733d2-85ad-4966-9740-97fffe7f54f6\" (UID: \"9d0733d2-85ad-4966-9740-97fffe7f54f6\") " Oct 30 18:41:35.105180 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:35.105154 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d0733d2-85ad-4966-9740-97fffe7f54f6-openshift-service-ca" pod "9d0733d2-85ad-4966-9740-97fffe7f54f6" (UID: "9d0733d2-85ad-4966-9740-97fffe7f54f6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:41:35.107064 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:35.107031 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d0733d2-85ad-4966-9740-97fffe7f54f6-kube-api-access-5qq2q" pod "9d0733d2-85ad-4966-9740-97fffe7f54f6" (UID: "9d0733d2-85ad-4966-9740-97fffe7f54f6"). InnerVolumeSpecName "kube-api-access-5qq2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:35.107469 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:35.107441 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d0733d2-85ad-4966-9740-97fffe7f54f6-metrics-certs" pod "9d0733d2-85ad-4966-9740-97fffe7f54f6" (UID: "9d0733d2-85ad-4966-9740-97fffe7f54f6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:41:35.107469 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:35.107460 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d0733d2-85ad-4966-9740-97fffe7f54f6-maas-api-tls" pod "9d0733d2-85ad-4966-9740-97fffe7f54f6" (UID: "9d0733d2-85ad-4966-9740-97fffe7f54f6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:41:35.206573 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:35.206534 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d0733d2-85ad-4966-9740-97fffe7f54f6-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:41:35.206573 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:35.206566 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d0733d2-85ad-4966-9740-97fffe7f54f6-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:41:35.206573 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:35.206579 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d0733d2-85ad-4966-9740-97fffe7f54f6-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:41:35.206822 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:35.206587 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qq2q\" (UniqueName: \"kubernetes.io/projected/9d0733d2-85ad-4966-9740-97fffe7f54f6-kube-api-access-5qq2q\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:41:35.761219 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:35.761185 4455 generic.go:356] "Generic (PLEG): container finished" podID="9d0733d2-85ad-4966-9740-97fffe7f54f6" containerID="8ed2643be90cff51f72e2ea02dfba6ff86cad80c591e675d79875db9b5f42f1e" exitCode=0 Oct 30 18:41:35.761461 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:35.761244 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4" Oct 30 18:41:35.761461 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:35.761265 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4" event={"ID":"9d0733d2-85ad-4966-9740-97fffe7f54f6","Type":"ContainerDied","Data":"8ed2643be90cff51f72e2ea02dfba6ff86cad80c591e675d79875db9b5f42f1e"} Oct 30 18:41:35.761461 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:35.761371 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4" event={"ID":"9d0733d2-85ad-4966-9740-97fffe7f54f6","Type":"ContainerDied","Data":"3cf887f97e49996f6122d9a7eff5a67c59dccb33f3da95b2935d1566647704cd"} Oct 30 18:41:35.761461 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:35.761407 4455 scope.go:122] "RemoveContainer" containerID="8ed2643be90cff51f72e2ea02dfba6ff86cad80c591e675d79875db9b5f42f1e" Oct 30 18:41:35.771580 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:35.771557 4455 scope.go:122] "RemoveContainer" containerID="8ed2643be90cff51f72e2ea02dfba6ff86cad80c591e675d79875db9b5f42f1e" Oct 30 18:41:35.771881 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:41:35.771859 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ed2643be90cff51f72e2ea02dfba6ff86cad80c591e675d79875db9b5f42f1e\": container with ID starting with 8ed2643be90cff51f72e2ea02dfba6ff86cad80c591e675d79875db9b5f42f1e not found: ID does not exist" containerID="8ed2643be90cff51f72e2ea02dfba6ff86cad80c591e675d79875db9b5f42f1e" Oct 30 18:41:35.771972 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:35.771893 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ed2643be90cff51f72e2ea02dfba6ff86cad80c591e675d79875db9b5f42f1e"} err="failed to get container status \"8ed2643be90cff51f72e2ea02dfba6ff86cad80c591e675d79875db9b5f42f1e\": rpc error: code = NotFound desc = could not find container \"8ed2643be90cff51f72e2ea02dfba6ff86cad80c591e675d79875db9b5f42f1e\": container with ID starting with 8ed2643be90cff51f72e2ea02dfba6ff86cad80c591e675d79875db9b5f42f1e not found: ID does not exist" Oct 30 18:41:35.816193 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:35.816146 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4"] Oct 30 18:41:35.825878 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:35.825833 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ac422f-64694584f7-d9pn4"] Oct 30 18:41:37.634960 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:41:37.634921 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d0733d2-85ad-4966-9740-97fffe7f54f6" path="/var/lib/kubelet/pods/9d0733d2-85ad-4966-9740-97fffe7f54f6/volumes" Oct 30 18:42:00.492347 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:42:00.489224 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:42:24.509256 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:24.509215 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs"] Oct 30 18:42:24.510904 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:24.509483 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" podUID="15651f51-8abe-47fa-89f8-a9b30ee0b3e9" containerName="istio-proxy" containerID="cri-o://d84286ac03dcb40d0c6ce75d2fa09e0bcde97fac943fb949f4924fdc17ce39ac" gracePeriod=30 Oct 30 18:42:29.791098 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:29.791040 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" Oct 30 18:42:29.858670 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:29.858635 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-workload-certs\") pod \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " Oct 30 18:42:29.859071 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:29.858769 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-credential-socket\") pod \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " Oct 30 18:42:29.859071 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:29.858802 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-istio-podinfo\") pod \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " Oct 30 18:42:29.859071 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:29.858876 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-istio-envoy\") pod \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " Oct 30 18:42:29.859071 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:29.858923 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-istio-token\" (UniqueName: \"kubernetes.io/projected/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-istio-token\") pod \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " Oct 30 18:42:29.859071 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:29.858927 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-workload-certs" pod "15651f51-8abe-47fa-89f8-a9b30ee0b3e9" (UID: "15651f51-8abe-47fa-89f8-a9b30ee0b3e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:42:29.859071 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:29.858957 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-istio-data\") pod \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " Oct 30 18:42:29.859071 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:29.858993 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-istiod-ca-cert\") pod \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " Oct 30 18:42:29.859071 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:29.859037 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-kube-api-access-bvbn7\" (UniqueName: \"kubernetes.io/projected/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-kube-api-access-bvbn7\") pod \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " Oct 30 18:42:29.860376 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:29.859090 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-workload-socket\") pod \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\" (UID: \"15651f51-8abe-47fa-89f8-a9b30ee0b3e9\") " Oct 30 18:42:29.860376 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:29.859574 4455 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-workload-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:42:29.860376 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:29.859833 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-workload-socket" pod "15651f51-8abe-47fa-89f8-a9b30ee0b3e9" (UID: "15651f51-8abe-47fa-89f8-a9b30ee0b3e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:42:29.860376 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:29.860111 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-istiod-ca-cert" pod "15651f51-8abe-47fa-89f8-a9b30ee0b3e9" (UID: "15651f51-8abe-47fa-89f8-a9b30ee0b3e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:42:29.860600 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:29.860395 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-istio-data" pod "15651f51-8abe-47fa-89f8-a9b30ee0b3e9" (UID: "15651f51-8abe-47fa-89f8-a9b30ee0b3e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:42:29.860864 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:29.860839 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-credential-socket" pod "15651f51-8abe-47fa-89f8-a9b30ee0b3e9" (UID: "15651f51-8abe-47fa-89f8-a9b30ee0b3e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:42:29.863059 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:29.863026 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-istio-podinfo" pod "15651f51-8abe-47fa-89f8-a9b30ee0b3e9" (UID: "15651f51-8abe-47fa-89f8-a9b30ee0b3e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:42:29.863274 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:29.863249 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-istio-token" pod "15651f51-8abe-47fa-89f8-a9b30ee0b3e9" (UID: "15651f51-8abe-47fa-89f8-a9b30ee0b3e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:42:29.863406 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:29.863338 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-kube-api-access-bvbn7" pod "15651f51-8abe-47fa-89f8-a9b30ee0b3e9" (UID: "15651f51-8abe-47fa-89f8-a9b30ee0b3e9"). InnerVolumeSpecName "kube-api-access-bvbn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:42:29.863578 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:29.863559 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-istio-envoy" pod "15651f51-8abe-47fa-89f8-a9b30ee0b3e9" (UID: "15651f51-8abe-47fa-89f8-a9b30ee0b3e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:42:29.960195 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:29.960165 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvbn7\" (UniqueName: \"kubernetes.io/projected/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-kube-api-access-bvbn7\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:42:29.960195 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:29.960195 4455 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-workload-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:42:29.960195 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:29.960204 4455 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-credential-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:42:29.960195 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:29.960214 4455 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-istio-podinfo\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:42:29.960584 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:29.960223 4455 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-istio-envoy\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:42:29.960584 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:29.960232 4455 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-istio-token\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:42:29.960584 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:29.960239 4455 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-istio-data\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:42:29.960584 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:29.960246 4455 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15651f51-8abe-47fa-89f8-a9b30ee0b3e9-istiod-ca-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:42:30.241771 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:30.241741 4455 generic.go:356] "Generic (PLEG): container finished" podID="15651f51-8abe-47fa-89f8-a9b30ee0b3e9" containerID="d84286ac03dcb40d0c6ce75d2fa09e0bcde97fac943fb949f4924fdc17ce39ac" exitCode=0 Oct 30 18:42:30.241961 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:30.241804 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" Oct 30 18:42:30.241961 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:30.241843 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" event={"ID":"15651f51-8abe-47fa-89f8-a9b30ee0b3e9","Type":"ContainerDied","Data":"d84286ac03dcb40d0c6ce75d2fa09e0bcde97fac943fb949f4924fdc17ce39ac"} Oct 30 18:42:30.241961 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:30.241901 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs" event={"ID":"15651f51-8abe-47fa-89f8-a9b30ee0b3e9","Type":"ContainerDied","Data":"713a6cc982202eeb9faa44cbd43be6f1a3d0d1c274fadaf8d4df4cef2115b237"} Oct 30 18:42:30.241961 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:30.241929 4455 scope.go:122] "RemoveContainer" containerID="d84286ac03dcb40d0c6ce75d2fa09e0bcde97fac943fb949f4924fdc17ce39ac" Oct 30 18:42:30.253216 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:30.253190 4455 scope.go:122] "RemoveContainer" containerID="d84286ac03dcb40d0c6ce75d2fa09e0bcde97fac943fb949f4924fdc17ce39ac" Oct 30 18:42:30.253577 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:42:30.253555 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d84286ac03dcb40d0c6ce75d2fa09e0bcde97fac943fb949f4924fdc17ce39ac\": container with ID starting with d84286ac03dcb40d0c6ce75d2fa09e0bcde97fac943fb949f4924fdc17ce39ac not found: ID does not exist" containerID="d84286ac03dcb40d0c6ce75d2fa09e0bcde97fac943fb949f4924fdc17ce39ac" Oct 30 18:42:30.253642 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:30.253586 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d84286ac03dcb40d0c6ce75d2fa09e0bcde97fac943fb949f4924fdc17ce39ac"} err="failed to get container status \"d84286ac03dcb40d0c6ce75d2fa09e0bcde97fac943fb949f4924fdc17ce39ac\": rpc error: code = NotFound desc = could not find container \"d84286ac03dcb40d0c6ce75d2fa09e0bcde97fac943fb949f4924fdc17ce39ac\": container with ID starting with d84286ac03dcb40d0c6ce75d2fa09e0bcde97fac943fb949f4924fdc17ce39ac not found: ID does not exist" Oct 30 18:42:30.293435 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:30.293403 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs"] Oct 30 18:42:30.304225 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:30.304194 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-nwtxs"] Oct 30 18:42:30.385252 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:42:30.385215 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:42:31.635497 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:31.635462 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15651f51-8abe-47fa-89f8-a9b30ee0b3e9" path="/var/lib/kubelet/pods/15651f51-8abe-47fa-89f8-a9b30ee0b3e9/volumes" Oct 30 18:42:35.698337 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:35.698284 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-293234-75cd6bf784-f78v8"] Oct 30 18:42:35.735691 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:35.735654 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-293234-b68cdd699-dcj4f"] Oct 30 18:42:39.884698 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:39.884590 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck"] Oct 30 18:42:39.886096 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:39.885815 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck" podUID="1524899f-95c7-4831-a90b-e108b715a3b3" containerName="maas-api" containerID="cri-o://b55cab462d6626fd11dbe4e012dabcd8995eea57c529b3755714f0a8b86eddea" gracePeriod=30 Oct 30 18:42:40.221545 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:40.221519 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck" Oct 30 18:42:40.329774 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:40.329741 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1524899f-95c7-4831-a90b-e108b715a3b3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1524899f-95c7-4831-a90b-e108b715a3b3-openshift-service-ca\") pod \"1524899f-95c7-4831-a90b-e108b715a3b3\" (UID: \"1524899f-95c7-4831-a90b-e108b715a3b3\") " Oct 30 18:42:40.330011 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:40.329890 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1524899f-95c7-4831-a90b-e108b715a3b3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1524899f-95c7-4831-a90b-e108b715a3b3-metrics-certs\") pod \"1524899f-95c7-4831-a90b-e108b715a3b3\" (UID: \"1524899f-95c7-4831-a90b-e108b715a3b3\") " Oct 30 18:42:40.330011 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:40.329950 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1524899f-95c7-4831-a90b-e108b715a3b3-kube-api-access-v86fs\" (UniqueName: \"kubernetes.io/projected/1524899f-95c7-4831-a90b-e108b715a3b3-kube-api-access-v86fs\") pod \"1524899f-95c7-4831-a90b-e108b715a3b3\" (UID: \"1524899f-95c7-4831-a90b-e108b715a3b3\") " Oct 30 18:42:40.330011 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:40.329968 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1524899f-95c7-4831-a90b-e108b715a3b3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1524899f-95c7-4831-a90b-e108b715a3b3-maas-api-tls\") pod \"1524899f-95c7-4831-a90b-e108b715a3b3\" (UID: \"1524899f-95c7-4831-a90b-e108b715a3b3\") " Oct 30 18:42:40.330235 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:40.330151 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1524899f-95c7-4831-a90b-e108b715a3b3-openshift-service-ca" pod "1524899f-95c7-4831-a90b-e108b715a3b3" (UID: "1524899f-95c7-4831-a90b-e108b715a3b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:42:40.330309 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:40.330297 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1524899f-95c7-4831-a90b-e108b715a3b3-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:42:40.332573 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:40.332505 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1524899f-95c7-4831-a90b-e108b715a3b3-kube-api-access-v86fs" pod "1524899f-95c7-4831-a90b-e108b715a3b3" (UID: "1524899f-95c7-4831-a90b-e108b715a3b3"). InnerVolumeSpecName "kube-api-access-v86fs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:42:40.332573 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:40.332510 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1524899f-95c7-4831-a90b-e108b715a3b3-maas-api-tls" pod "1524899f-95c7-4831-a90b-e108b715a3b3" (UID: "1524899f-95c7-4831-a90b-e108b715a3b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:42:40.332767 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:40.332585 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1524899f-95c7-4831-a90b-e108b715a3b3-metrics-certs" pod "1524899f-95c7-4831-a90b-e108b715a3b3" (UID: "1524899f-95c7-4831-a90b-e108b715a3b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:42:40.339149 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:40.339119 4455 generic.go:356] "Generic (PLEG): container finished" podID="1524899f-95c7-4831-a90b-e108b715a3b3" containerID="b55cab462d6626fd11dbe4e012dabcd8995eea57c529b3755714f0a8b86eddea" exitCode=0 Oct 30 18:42:40.339286 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:40.339160 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck" Oct 30 18:42:40.339286 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:40.339216 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck" event={"ID":"1524899f-95c7-4831-a90b-e108b715a3b3","Type":"ContainerDied","Data":"b55cab462d6626fd11dbe4e012dabcd8995eea57c529b3755714f0a8b86eddea"} Oct 30 18:42:40.339286 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:40.339282 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck" event={"ID":"1524899f-95c7-4831-a90b-e108b715a3b3","Type":"ContainerDied","Data":"2217506b916a99e19113b72e6187e0b28046991448592b663e394412235a352b"} Oct 30 18:42:40.339446 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:40.339311 4455 scope.go:122] "RemoveContainer" containerID="b55cab462d6626fd11dbe4e012dabcd8995eea57c529b3755714f0a8b86eddea" Oct 30 18:42:40.361227 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:40.361200 4455 scope.go:122] "RemoveContainer" containerID="b55cab462d6626fd11dbe4e012dabcd8995eea57c529b3755714f0a8b86eddea" Oct 30 18:42:40.361695 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:42:40.361674 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b55cab462d6626fd11dbe4e012dabcd8995eea57c529b3755714f0a8b86eddea\": container with ID starting with b55cab462d6626fd11dbe4e012dabcd8995eea57c529b3755714f0a8b86eddea not found: ID does not exist" containerID="b55cab462d6626fd11dbe4e012dabcd8995eea57c529b3755714f0a8b86eddea" Oct 30 18:42:40.361751 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:40.361707 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b55cab462d6626fd11dbe4e012dabcd8995eea57c529b3755714f0a8b86eddea"} err="failed to get container status \"b55cab462d6626fd11dbe4e012dabcd8995eea57c529b3755714f0a8b86eddea\": rpc error: code = NotFound desc = could not find container \"b55cab462d6626fd11dbe4e012dabcd8995eea57c529b3755714f0a8b86eddea\": container with ID starting with b55cab462d6626fd11dbe4e012dabcd8995eea57c529b3755714f0a8b86eddea not found: ID does not exist" Oct 30 18:42:40.393590 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:40.393555 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck"] Oct 30 18:42:40.398108 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:40.398081 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-293234-68b67c8cf-7hhck"] Oct 30 18:42:40.431658 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:40.431617 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1524899f-95c7-4831-a90b-e108b715a3b3-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:42:40.431658 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:40.431648 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v86fs\" (UniqueName: \"kubernetes.io/projected/1524899f-95c7-4831-a90b-e108b715a3b3-kube-api-access-v86fs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:42:40.431658 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:40.431658 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1524899f-95c7-4831-a90b-e108b715a3b3-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:42:40.650244 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:40.650210 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs"] Oct 30 18:42:40.650543 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:40.650495 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs" podUID="96e42878-31cc-4d86-b428-e39a8aa72218" containerName="discovery" containerID="cri-o://88ca55b7819e9a1e970b04be6c0d19e087497312ad24e57a9b20dec92345509f" gracePeriod=30 Oct 30 18:42:40.698848 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:40.698801 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-293234-75cd6bf784-f78v8" podUID="fc61777b-475d-4137-91eb-56e488ece861" containerName="payload-processing" containerID="cri-o://7a3b9548e4d3e650b112866b33f876c355038dc134470904e5084534b5ce00ca" gracePeriod=25 Oct 30 18:42:40.736117 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:40.736032 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-293234-b68cdd699-dcj4f" podUID="877c2eae-69e9-42fe-bbb9-b7a8ced3fa06" containerName="payload-pre-processing" containerID="cri-o://8e6da89cad7efa940e420d81b1d841d83909a28ada3a01e72ec9fc267a9f0937" gracePeriod=25 Oct 30 18:42:41.014444 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.014411 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs" Oct 30 18:42:41.057032 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.057005 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-293234-75cd6bf784-f78v8" Oct 30 18:42:41.083981 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.083953 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-293234-b68cdd699-dcj4f" Oct 30 18:42:41.143389 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.143349 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96e42878-31cc-4d86-b428-e39a8aa72218-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/96e42878-31cc-4d86-b428-e39a8aa72218-local-certs\") pod \"96e42878-31cc-4d86-b428-e39a8aa72218\" (UID: \"96e42878-31cc-4d86-b428-e39a8aa72218\") " Oct 30 18:42:41.143621 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.143415 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc61777b-475d-4137-91eb-56e488ece861-kube-api-access-rxxnn\" (UniqueName: \"kubernetes.io/projected/fc61777b-475d-4137-91eb-56e488ece861-kube-api-access-rxxnn\") pod \"fc61777b-475d-4137-91eb-56e488ece861\" (UID: \"fc61777b-475d-4137-91eb-56e488ece861\") " Oct 30 18:42:41.143621 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.143469 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96e42878-31cc-4d86-b428-e39a8aa72218-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/96e42878-31cc-4d86-b428-e39a8aa72218-istio-kubeconfig\") pod \"96e42878-31cc-4d86-b428-e39a8aa72218\" (UID: \"96e42878-31cc-4d86-b428-e39a8aa72218\") " Oct 30 18:42:41.143621 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.143560 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96e42878-31cc-4d86-b428-e39a8aa72218-istio-token\" (UniqueName: \"kubernetes.io/projected/96e42878-31cc-4d86-b428-e39a8aa72218-istio-token\") pod \"96e42878-31cc-4d86-b428-e39a8aa72218\" (UID: \"96e42878-31cc-4d86-b428-e39a8aa72218\") " Oct 30 18:42:41.143621 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.143606 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96e42878-31cc-4d86-b428-e39a8aa72218-kube-api-access-6jm7w\" (UniqueName: \"kubernetes.io/projected/96e42878-31cc-4d86-b428-e39a8aa72218-kube-api-access-6jm7w\") pod \"96e42878-31cc-4d86-b428-e39a8aa72218\" (UID: \"96e42878-31cc-4d86-b428-e39a8aa72218\") " Oct 30 18:42:41.143873 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.143675 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc61777b-475d-4137-91eb-56e488ece861-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc61777b-475d-4137-91eb-56e488ece861-plugins-config-volume\") pod \"fc61777b-475d-4137-91eb-56e488ece861\" (UID: \"fc61777b-475d-4137-91eb-56e488ece861\") " Oct 30 18:42:41.143873 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.143727 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96e42878-31cc-4d86-b428-e39a8aa72218-cacerts\" (UniqueName: \"kubernetes.io/secret/96e42878-31cc-4d86-b428-e39a8aa72218-cacerts\") pod \"96e42878-31cc-4d86-b428-e39a8aa72218\" (UID: \"96e42878-31cc-4d86-b428-e39a8aa72218\") " Oct 30 18:42:41.143873 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.143757 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96e42878-31cc-4d86-b428-e39a8aa72218-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/96e42878-31cc-4d86-b428-e39a8aa72218-istio-csr-dns-cert\") pod \"96e42878-31cc-4d86-b428-e39a8aa72218\" (UID: \"96e42878-31cc-4d86-b428-e39a8aa72218\") " Oct 30 18:42:41.143873 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.143799 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/877c2eae-69e9-42fe-bbb9-b7a8ced3fa06-kube-api-access-ttzzh\" (UniqueName: \"kubernetes.io/projected/877c2eae-69e9-42fe-bbb9-b7a8ced3fa06-kube-api-access-ttzzh\") pod \"877c2eae-69e9-42fe-bbb9-b7a8ced3fa06\" (UID: \"877c2eae-69e9-42fe-bbb9-b7a8ced3fa06\") " Oct 30 18:42:41.144069 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.143875 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/877c2eae-69e9-42fe-bbb9-b7a8ced3fa06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/877c2eae-69e9-42fe-bbb9-b7a8ced3fa06-plugins-config-volume\") pod \"877c2eae-69e9-42fe-bbb9-b7a8ced3fa06\" (UID: \"877c2eae-69e9-42fe-bbb9-b7a8ced3fa06\") " Oct 30 18:42:41.144069 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.143922 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96e42878-31cc-4d86-b428-e39a8aa72218-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/96e42878-31cc-4d86-b428-e39a8aa72218-istio-csr-ca-configmap\") pod \"96e42878-31cc-4d86-b428-e39a8aa72218\" (UID: \"96e42878-31cc-4d86-b428-e39a8aa72218\") " Oct 30 18:42:41.144591 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.144304 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc61777b-475d-4137-91eb-56e488ece861-plugins-config-volume" pod "fc61777b-475d-4137-91eb-56e488ece861" (UID: "fc61777b-475d-4137-91eb-56e488ece861"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:42:41.144591 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.144461 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc61777b-475d-4137-91eb-56e488ece861-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:42:41.144771 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.144736 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/877c2eae-69e9-42fe-bbb9-b7a8ced3fa06-plugins-config-volume" pod "877c2eae-69e9-42fe-bbb9-b7a8ced3fa06" (UID: "877c2eae-69e9-42fe-bbb9-b7a8ced3fa06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:42:41.146586 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.146532 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96e42878-31cc-4d86-b428-e39a8aa72218-istio-csr-ca-configmap" pod "96e42878-31cc-4d86-b428-e39a8aa72218" (UID: "96e42878-31cc-4d86-b428-e39a8aa72218"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:42:41.146736 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.146620 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e42878-31cc-4d86-b428-e39a8aa72218-local-certs" pod "96e42878-31cc-4d86-b428-e39a8aa72218" (UID: "96e42878-31cc-4d86-b428-e39a8aa72218"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:42:41.148460 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.148428 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96e42878-31cc-4d86-b428-e39a8aa72218-cacerts" pod "96e42878-31cc-4d86-b428-e39a8aa72218" (UID: "96e42878-31cc-4d86-b428-e39a8aa72218"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:42:41.148743 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.148721 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96e42878-31cc-4d86-b428-e39a8aa72218-istio-csr-dns-cert" pod "96e42878-31cc-4d86-b428-e39a8aa72218" (UID: "96e42878-31cc-4d86-b428-e39a8aa72218"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:42:41.149031 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.148996 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96e42878-31cc-4d86-b428-e39a8aa72218-istio-kubeconfig" pod "96e42878-31cc-4d86-b428-e39a8aa72218" (UID: "96e42878-31cc-4d86-b428-e39a8aa72218"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:42:41.149031 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.149009 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/877c2eae-69e9-42fe-bbb9-b7a8ced3fa06-kube-api-access-ttzzh" pod "877c2eae-69e9-42fe-bbb9-b7a8ced3fa06" (UID: "877c2eae-69e9-42fe-bbb9-b7a8ced3fa06"). InnerVolumeSpecName "kube-api-access-ttzzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:42:41.149400 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.149375 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96e42878-31cc-4d86-b428-e39a8aa72218-istio-token" pod "96e42878-31cc-4d86-b428-e39a8aa72218" (UID: "96e42878-31cc-4d86-b428-e39a8aa72218"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:42:41.149519 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.149423 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc61777b-475d-4137-91eb-56e488ece861-kube-api-access-rxxnn" pod "fc61777b-475d-4137-91eb-56e488ece861" (UID: "fc61777b-475d-4137-91eb-56e488ece861"). InnerVolumeSpecName "kube-api-access-rxxnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:42:41.149519 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.149483 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96e42878-31cc-4d86-b428-e39a8aa72218-kube-api-access-6jm7w" pod "96e42878-31cc-4d86-b428-e39a8aa72218" (UID: "96e42878-31cc-4d86-b428-e39a8aa72218"). InnerVolumeSpecName "kube-api-access-6jm7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:42:41.245357 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.245285 4455 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/96e42878-31cc-4d86-b428-e39a8aa72218-cacerts\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:42:41.245357 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.245349 4455 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/96e42878-31cc-4d86-b428-e39a8aa72218-istio-csr-dns-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:42:41.245357 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.245363 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttzzh\" (UniqueName: \"kubernetes.io/projected/877c2eae-69e9-42fe-bbb9-b7a8ced3fa06-kube-api-access-ttzzh\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:42:41.245633 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.245374 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/877c2eae-69e9-42fe-bbb9-b7a8ced3fa06-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:42:41.245633 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.245384 4455 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/96e42878-31cc-4d86-b428-e39a8aa72218-istio-csr-ca-configmap\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:42:41.245633 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.245392 4455 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/96e42878-31cc-4d86-b428-e39a8aa72218-local-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:42:41.245633 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.245401 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxxnn\" (UniqueName: \"kubernetes.io/projected/fc61777b-475d-4137-91eb-56e488ece861-kube-api-access-rxxnn\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:42:41.245633 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.245410 4455 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/96e42878-31cc-4d86-b428-e39a8aa72218-istio-kubeconfig\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:42:41.245633 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.245417 4455 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96e42878-31cc-4d86-b428-e39a8aa72218-istio-token\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:42:41.245633 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.245425 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jm7w\" (UniqueName: \"kubernetes.io/projected/96e42878-31cc-4d86-b428-e39a8aa72218-kube-api-access-6jm7w\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:42:41.349352 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.349304 4455 generic.go:356] "Generic (PLEG): container finished" podID="877c2eae-69e9-42fe-bbb9-b7a8ced3fa06" containerID="8e6da89cad7efa940e420d81b1d841d83909a28ada3a01e72ec9fc267a9f0937" exitCode=0 Oct 30 18:42:41.349557 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.349382 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-293234-b68cdd699-dcj4f" Oct 30 18:42:41.349557 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.349395 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-293234-b68cdd699-dcj4f" event={"ID":"877c2eae-69e9-42fe-bbb9-b7a8ced3fa06","Type":"ContainerDied","Data":"8e6da89cad7efa940e420d81b1d841d83909a28ada3a01e72ec9fc267a9f0937"} Oct 30 18:42:41.349557 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.349498 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-293234-b68cdd699-dcj4f" event={"ID":"877c2eae-69e9-42fe-bbb9-b7a8ced3fa06","Type":"ContainerDied","Data":"37a6bf4f25da988b9722357105847a47bfd79d0b616bf105f4d31b0284888554"} Oct 30 18:42:41.349557 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.349530 4455 scope.go:122] "RemoveContainer" containerID="8e6da89cad7efa940e420d81b1d841d83909a28ada3a01e72ec9fc267a9f0937" Oct 30 18:42:41.351941 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.351917 4455 generic.go:356] "Generic (PLEG): container finished" podID="fc61777b-475d-4137-91eb-56e488ece861" containerID="7a3b9548e4d3e650b112866b33f876c355038dc134470904e5084534b5ce00ca" exitCode=0 Oct 30 18:42:41.352060 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.351968 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-293234-75cd6bf784-f78v8" Oct 30 18:42:41.352060 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.351968 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-293234-75cd6bf784-f78v8" event={"ID":"fc61777b-475d-4137-91eb-56e488ece861","Type":"ContainerDied","Data":"7a3b9548e4d3e650b112866b33f876c355038dc134470904e5084534b5ce00ca"} Oct 30 18:42:41.352179 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.352087 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-293234-75cd6bf784-f78v8" event={"ID":"fc61777b-475d-4137-91eb-56e488ece861","Type":"ContainerDied","Data":"f5c533cafef578ab4800395664aa31522b0942880fb95738c1cf47fe8e88773e"} Oct 30 18:42:41.353633 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.353609 4455 generic.go:356] "Generic (PLEG): container finished" podID="96e42878-31cc-4d86-b428-e39a8aa72218" containerID="88ca55b7819e9a1e970b04be6c0d19e087497312ad24e57a9b20dec92345509f" exitCode=0 Oct 30 18:42:41.353785 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.353642 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs" Oct 30 18:42:41.353785 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.353682 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs" event={"ID":"96e42878-31cc-4d86-b428-e39a8aa72218","Type":"ContainerDied","Data":"88ca55b7819e9a1e970b04be6c0d19e087497312ad24e57a9b20dec92345509f"} Oct 30 18:42:41.353785 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.353723 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs" event={"ID":"96e42878-31cc-4d86-b428-e39a8aa72218","Type":"ContainerDied","Data":"3e9f8bc9ec22891fbb476505e1295e5f52568930395db12b7e53882fe56943fa"} Oct 30 18:42:41.363137 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.363114 4455 scope.go:122] "RemoveContainer" containerID="8e6da89cad7efa940e420d81b1d841d83909a28ada3a01e72ec9fc267a9f0937" Oct 30 18:42:41.363447 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:42:41.363428 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e6da89cad7efa940e420d81b1d841d83909a28ada3a01e72ec9fc267a9f0937\": container with ID starting with 8e6da89cad7efa940e420d81b1d841d83909a28ada3a01e72ec9fc267a9f0937 not found: ID does not exist" containerID="8e6da89cad7efa940e420d81b1d841d83909a28ada3a01e72ec9fc267a9f0937" Oct 30 18:42:41.363542 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.363455 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e6da89cad7efa940e420d81b1d841d83909a28ada3a01e72ec9fc267a9f0937"} err="failed to get container status \"8e6da89cad7efa940e420d81b1d841d83909a28ada3a01e72ec9fc267a9f0937\": rpc error: code = NotFound desc = could not find container \"8e6da89cad7efa940e420d81b1d841d83909a28ada3a01e72ec9fc267a9f0937\": container with ID starting with 8e6da89cad7efa940e420d81b1d841d83909a28ada3a01e72ec9fc267a9f0937 not found: ID does not exist" Oct 30 18:42:41.363542 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.363473 4455 scope.go:122] "RemoveContainer" containerID="7a3b9548e4d3e650b112866b33f876c355038dc134470904e5084534b5ce00ca" Oct 30 18:42:41.375432 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.375404 4455 scope.go:122] "RemoveContainer" containerID="7a3b9548e4d3e650b112866b33f876c355038dc134470904e5084534b5ce00ca" Oct 30 18:42:41.375804 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:42:41.375782 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a3b9548e4d3e650b112866b33f876c355038dc134470904e5084534b5ce00ca\": container with ID starting with 7a3b9548e4d3e650b112866b33f876c355038dc134470904e5084534b5ce00ca not found: ID does not exist" containerID="7a3b9548e4d3e650b112866b33f876c355038dc134470904e5084534b5ce00ca" Oct 30 18:42:41.375897 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.375812 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a3b9548e4d3e650b112866b33f876c355038dc134470904e5084534b5ce00ca"} err="failed to get container status \"7a3b9548e4d3e650b112866b33f876c355038dc134470904e5084534b5ce00ca\": rpc error: code = NotFound desc = could not find container \"7a3b9548e4d3e650b112866b33f876c355038dc134470904e5084534b5ce00ca\": container with ID starting with 7a3b9548e4d3e650b112866b33f876c355038dc134470904e5084534b5ce00ca not found: ID does not exist" Oct 30 18:42:41.375897 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.375832 4455 scope.go:122] "RemoveContainer" containerID="88ca55b7819e9a1e970b04be6c0d19e087497312ad24e57a9b20dec92345509f" Oct 30 18:42:41.385642 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.385617 4455 scope.go:122] "RemoveContainer" containerID="88ca55b7819e9a1e970b04be6c0d19e087497312ad24e57a9b20dec92345509f" Oct 30 18:42:41.386023 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:42:41.385998 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88ca55b7819e9a1e970b04be6c0d19e087497312ad24e57a9b20dec92345509f\": container with ID starting with 88ca55b7819e9a1e970b04be6c0d19e087497312ad24e57a9b20dec92345509f not found: ID does not exist" containerID="88ca55b7819e9a1e970b04be6c0d19e087497312ad24e57a9b20dec92345509f" Oct 30 18:42:41.386105 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.386032 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88ca55b7819e9a1e970b04be6c0d19e087497312ad24e57a9b20dec92345509f"} err="failed to get container status \"88ca55b7819e9a1e970b04be6c0d19e087497312ad24e57a9b20dec92345509f\": rpc error: code = NotFound desc = could not find container \"88ca55b7819e9a1e970b04be6c0d19e087497312ad24e57a9b20dec92345509f\": container with ID starting with 88ca55b7819e9a1e970b04be6c0d19e087497312ad24e57a9b20dec92345509f not found: ID does not exist" Oct 30 18:42:41.431769 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.431734 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs"] Oct 30 18:42:41.445992 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.445956 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z2svs"] Oct 30 18:42:41.481244 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.481209 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-293234-b68cdd699-dcj4f"] Oct 30 18:42:41.486682 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.486645 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-293234-b68cdd699-dcj4f"] Oct 30 18:42:41.518874 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.518791 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-293234-75cd6bf784-f78v8"] Oct 30 18:42:41.525927 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.525891 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-293234-75cd6bf784-f78v8"] Oct 30 18:42:41.633604 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.633571 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1524899f-95c7-4831-a90b-e108b715a3b3" path="/var/lib/kubelet/pods/1524899f-95c7-4831-a90b-e108b715a3b3/volumes" Oct 30 18:42:41.634006 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.633993 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="877c2eae-69e9-42fe-bbb9-b7a8ced3fa06" path="/var/lib/kubelet/pods/877c2eae-69e9-42fe-bbb9-b7a8ced3fa06/volumes" Oct 30 18:42:41.634470 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.634443 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96e42878-31cc-4d86-b428-e39a8aa72218" path="/var/lib/kubelet/pods/96e42878-31cc-4d86-b428-e39a8aa72218/volumes" Oct 30 18:42:41.635248 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:42:41.635230 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc61777b-475d-4137-91eb-56e488ece861" path="/var/lib/kubelet/pods/fc61777b-475d-4137-91eb-56e488ece861/volumes" Oct 30 18:43:00.385615 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:43:00.385568 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:43:30.385825 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:43:30.385785 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:44:00.403068 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:44:00.402779 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:44:12.637135 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:12.637030 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b424a5-b698d4bb8-vvqmn"] Oct 30 18:44:12.674971 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:12.674943 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b424a5-7d97cc8c87-vvjcw"] Oct 30 18:44:17.637917 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:17.637875 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-b424a5-b698d4bb8-vvqmn" podUID="61033891-2264-449f-9a4b-d5b775d51ca6" containerName="payload-processing" containerID="cri-o://6876f31cc94d72383d5346128cd01f57e4cfd518824d39c00be506c9c15a4bc8" gracePeriod=25 Oct 30 18:44:17.676447 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:17.676164 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b424a5-7d97cc8c87-vvjcw" podUID="1e050532-86d8-4954-ae7a-b0fcd484ad24" containerName="payload-pre-processing" containerID="cri-o://eeda27f88a2295a5daf43a0093fc91d5b02d804fd8aec3821f95168389ec5491" gracePeriod=25 Oct 30 18:44:18.019525 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:18.019466 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-b424a5-b698d4bb8-vvqmn" podUID="61033891-2264-449f-9a4b-d5b775d51ca6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:44:26.006348 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:26.005422 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b424a5-7d97cc8c87-vvjcw" podUID="1e050532-86d8-4954-ae7a-b0fcd484ad24" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:44:28.013705 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:28.013667 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-b424a5-b698d4bb8-vvqmn" podUID="61033891-2264-449f-9a4b-d5b775d51ca6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:44:30.368452 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:30.368417 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n"] Oct 30 18:44:30.368900 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:30.368649 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n" podUID="dbdbb8b6-80ee-4abe-990e-72e4add7e77b" containerName="maas-api" containerID="cri-o://316a88c2717a8b365e51913c51d1d1935b2e763c6c265d44ed52ae48312f3001" gracePeriod=30 Oct 30 18:44:30.406875 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:44:30.406834 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:44:30.688233 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:30.688198 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n" Oct 30 18:44:30.835780 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:30.835734 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbdbb8b6-80ee-4abe-990e-72e4add7e77b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbdbb8b6-80ee-4abe-990e-72e4add7e77b-maas-api-tls\") pod \"dbdbb8b6-80ee-4abe-990e-72e4add7e77b\" (UID: \"dbdbb8b6-80ee-4abe-990e-72e4add7e77b\") " Oct 30 18:44:30.835996 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:30.835811 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbdbb8b6-80ee-4abe-990e-72e4add7e77b-kube-api-access-rbtlw\" (UniqueName: \"kubernetes.io/projected/dbdbb8b6-80ee-4abe-990e-72e4add7e77b-kube-api-access-rbtlw\") pod \"dbdbb8b6-80ee-4abe-990e-72e4add7e77b\" (UID: \"dbdbb8b6-80ee-4abe-990e-72e4add7e77b\") " Oct 30 18:44:30.835996 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:30.835925 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbdbb8b6-80ee-4abe-990e-72e4add7e77b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbdbb8b6-80ee-4abe-990e-72e4add7e77b-openshift-service-ca\") pod \"dbdbb8b6-80ee-4abe-990e-72e4add7e77b\" (UID: \"dbdbb8b6-80ee-4abe-990e-72e4add7e77b\") " Oct 30 18:44:30.836138 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:30.836023 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbdbb8b6-80ee-4abe-990e-72e4add7e77b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbdbb8b6-80ee-4abe-990e-72e4add7e77b-metrics-certs\") pod \"dbdbb8b6-80ee-4abe-990e-72e4add7e77b\" (UID: \"dbdbb8b6-80ee-4abe-990e-72e4add7e77b\") " Oct 30 18:44:30.849897 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:30.849851 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbdbb8b6-80ee-4abe-990e-72e4add7e77b-openshift-service-ca" pod "dbdbb8b6-80ee-4abe-990e-72e4add7e77b" (UID: "dbdbb8b6-80ee-4abe-990e-72e4add7e77b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:30.855702 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:30.854234 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbdbb8b6-80ee-4abe-990e-72e4add7e77b-kube-api-access-rbtlw" pod "dbdbb8b6-80ee-4abe-990e-72e4add7e77b" (UID: "dbdbb8b6-80ee-4abe-990e-72e4add7e77b"). InnerVolumeSpecName "kube-api-access-rbtlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:30.856058 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:30.856026 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbdbb8b6-80ee-4abe-990e-72e4add7e77b-metrics-certs" pod "dbdbb8b6-80ee-4abe-990e-72e4add7e77b" (UID: "dbdbb8b6-80ee-4abe-990e-72e4add7e77b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:44:30.856357 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:30.856302 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbdbb8b6-80ee-4abe-990e-72e4add7e77b-maas-api-tls" pod "dbdbb8b6-80ee-4abe-990e-72e4add7e77b" (UID: "dbdbb8b6-80ee-4abe-990e-72e4add7e77b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:44:30.939390 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:30.937208 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbdbb8b6-80ee-4abe-990e-72e4add7e77b-maas-api-tls\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:44:30.939390 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:30.937243 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbtlw\" (UniqueName: \"kubernetes.io/projected/dbdbb8b6-80ee-4abe-990e-72e4add7e77b-kube-api-access-rbtlw\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:44:30.939390 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:30.937260 4455 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbdbb8b6-80ee-4abe-990e-72e4add7e77b-openshift-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:44:30.939390 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:30.937273 4455 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbdbb8b6-80ee-4abe-990e-72e4add7e77b-metrics-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:44:31.374960 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:31.374929 4455 generic.go:356] "Generic (PLEG): container finished" podID="dbdbb8b6-80ee-4abe-990e-72e4add7e77b" containerID="316a88c2717a8b365e51913c51d1d1935b2e763c6c265d44ed52ae48312f3001" exitCode=0 Oct 30 18:44:31.375583 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:31.374978 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n" Oct 30 18:44:31.375583 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:31.375004 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n" event={"ID":"dbdbb8b6-80ee-4abe-990e-72e4add7e77b","Type":"ContainerDied","Data":"316a88c2717a8b365e51913c51d1d1935b2e763c6c265d44ed52ae48312f3001"} Oct 30 18:44:31.375583 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:31.375073 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n" event={"ID":"dbdbb8b6-80ee-4abe-990e-72e4add7e77b","Type":"ContainerDied","Data":"206be537d45ae691981b5915fb6b83257f06c29dd5b37e404de013159cc99eb1"} Oct 30 18:44:31.375583 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:31.375102 4455 scope.go:122] "RemoveContainer" containerID="316a88c2717a8b365e51913c51d1d1935b2e763c6c265d44ed52ae48312f3001" Oct 30 18:44:31.395081 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:31.395063 4455 scope.go:122] "RemoveContainer" containerID="316a88c2717a8b365e51913c51d1d1935b2e763c6c265d44ed52ae48312f3001" Oct 30 18:44:31.395425 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:44:31.395395 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"316a88c2717a8b365e51913c51d1d1935b2e763c6c265d44ed52ae48312f3001\": container with ID starting with 316a88c2717a8b365e51913c51d1d1935b2e763c6c265d44ed52ae48312f3001 not found: ID does not exist" containerID="316a88c2717a8b365e51913c51d1d1935b2e763c6c265d44ed52ae48312f3001" Oct 30 18:44:31.395522 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:31.395432 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"316a88c2717a8b365e51913c51d1d1935b2e763c6c265d44ed52ae48312f3001"} err="failed to get container status \"316a88c2717a8b365e51913c51d1d1935b2e763c6c265d44ed52ae48312f3001\": rpc error: code = NotFound desc = could not find container \"316a88c2717a8b365e51913c51d1d1935b2e763c6c265d44ed52ae48312f3001\": container with ID starting with 316a88c2717a8b365e51913c51d1d1935b2e763c6c265d44ed52ae48312f3001 not found: ID does not exist" Oct 30 18:44:31.429463 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:31.429429 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n"] Oct 30 18:44:31.440354 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:31.440304 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b424a5-69c648d8fc-bk65n"] Oct 30 18:44:31.633587 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:31.633494 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbdbb8b6-80ee-4abe-990e-72e4add7e77b" path="/var/lib/kubelet/pods/dbdbb8b6-80ee-4abe-990e-72e4add7e77b/volumes" Oct 30 18:44:35.993160 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:35.993121 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b424a5-7d97cc8c87-vvjcw" podUID="1e050532-86d8-4954-ae7a-b0fcd484ad24" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:44:37.819365 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:37.819095 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b424a5-b698d4bb8-vvqmn" Oct 30 18:44:37.863357 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:37.862169 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd"] Oct 30 18:44:37.863357 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:37.862451 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" podUID="1d3471dc-aaf3-4060-a64d-2413e522e65a" containerName="istio-proxy" containerID="cri-o://af56ed4e6bb5f101fe282f280b2506a9145a50eefe1f248c8a104b25b4ae0d67" gracePeriod=30 Oct 30 18:44:37.868171 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:37.866190 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx"] Oct 30 18:44:37.868171 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:37.866392 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" podUID="4e7b0c73-0041-4549-b47e-76a7aaed2b56" containerName="istio-proxy" containerID="cri-o://79d17f879566861647cf69447a10eeff41d3d2ff0ccd53c84938d1107672ab0f" gracePeriod=30 Oct 30 18:44:37.899767 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:37.899674 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx"] Oct 30 18:44:37.901434 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:37.900296 4455 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" podUID="c7c21be6-b9eb-401c-aa3c-af5b62960cd8" containerName="istio-proxy" containerID="cri-o://2e57522cd1c5af27c444736562b4275230f8fe90ef6a33f7e7ef157c5cc09714" gracePeriod=30 Oct 30 18:44:37.919215 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:37.917801 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61033891-2264-449f-9a4b-d5b775d51ca6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61033891-2264-449f-9a4b-d5b775d51ca6-plugins-config-volume\") pod \"61033891-2264-449f-9a4b-d5b775d51ca6\" (UID: \"61033891-2264-449f-9a4b-d5b775d51ca6\") " Oct 30 18:44:37.919215 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:37.917927 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61033891-2264-449f-9a4b-d5b775d51ca6-kube-api-access-mc8pg\" (UniqueName: \"kubernetes.io/projected/61033891-2264-449f-9a4b-d5b775d51ca6-kube-api-access-mc8pg\") pod \"61033891-2264-449f-9a4b-d5b775d51ca6\" (UID: \"61033891-2264-449f-9a4b-d5b775d51ca6\") " Oct 30 18:44:37.919215 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:37.919012 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61033891-2264-449f-9a4b-d5b775d51ca6-plugins-config-volume" pod "61033891-2264-449f-9a4b-d5b775d51ca6" (UID: "61033891-2264-449f-9a4b-d5b775d51ca6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:37.929393 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:37.925493 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61033891-2264-449f-9a4b-d5b775d51ca6-kube-api-access-mc8pg" pod "61033891-2264-449f-9a4b-d5b775d51ca6" (UID: "61033891-2264-449f-9a4b-d5b775d51ca6"). InnerVolumeSpecName "kube-api-access-mc8pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:38.015139 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:38.015106 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b424a5-7d97cc8c87-vvjcw" Oct 30 18:44:38.019102 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:38.019075 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61033891-2264-449f-9a4b-d5b775d51ca6-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:44:38.019236 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:38.019109 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc8pg\" (UniqueName: \"kubernetes.io/projected/61033891-2264-449f-9a4b-d5b775d51ca6-kube-api-access-mc8pg\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:44:38.119761 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:38.119723 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e050532-86d8-4954-ae7a-b0fcd484ad24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e050532-86d8-4954-ae7a-b0fcd484ad24-plugins-config-volume\") pod \"1e050532-86d8-4954-ae7a-b0fcd484ad24\" (UID: \"1e050532-86d8-4954-ae7a-b0fcd484ad24\") " Oct 30 18:44:38.119955 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:38.119813 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e050532-86d8-4954-ae7a-b0fcd484ad24-kube-api-access-pwl6q\" (UniqueName: \"kubernetes.io/projected/1e050532-86d8-4954-ae7a-b0fcd484ad24-kube-api-access-pwl6q\") pod \"1e050532-86d8-4954-ae7a-b0fcd484ad24\" (UID: \"1e050532-86d8-4954-ae7a-b0fcd484ad24\") " Oct 30 18:44:38.120130 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:38.120105 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e050532-86d8-4954-ae7a-b0fcd484ad24-plugins-config-volume" pod "1e050532-86d8-4954-ae7a-b0fcd484ad24" (UID: "1e050532-86d8-4954-ae7a-b0fcd484ad24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:38.120258 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:38.120240 4455 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e050532-86d8-4954-ae7a-b0fcd484ad24-plugins-config-volume\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:44:38.123638 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:38.122562 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e050532-86d8-4954-ae7a-b0fcd484ad24-kube-api-access-pwl6q" pod "1e050532-86d8-4954-ae7a-b0fcd484ad24" (UID: "1e050532-86d8-4954-ae7a-b0fcd484ad24"). InnerVolumeSpecName "kube-api-access-pwl6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:38.220988 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:38.220957 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwl6q\" (UniqueName: \"kubernetes.io/projected/1e050532-86d8-4954-ae7a-b0fcd484ad24-kube-api-access-pwl6q\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:44:38.417058 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:38.417017 4455 generic.go:356] "Generic (PLEG): container finished" podID="61033891-2264-449f-9a4b-d5b775d51ca6" containerID="6876f31cc94d72383d5346128cd01f57e4cfd518824d39c00be506c9c15a4bc8" exitCode=0 Oct 30 18:44:38.417253 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:38.417080 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b424a5-b698d4bb8-vvqmn" Oct 30 18:44:38.417253 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:38.417102 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b424a5-b698d4bb8-vvqmn" event={"ID":"61033891-2264-449f-9a4b-d5b775d51ca6","Type":"ContainerDied","Data":"6876f31cc94d72383d5346128cd01f57e4cfd518824d39c00be506c9c15a4bc8"} Oct 30 18:44:38.417253 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:38.417161 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b424a5-b698d4bb8-vvqmn" event={"ID":"61033891-2264-449f-9a4b-d5b775d51ca6","Type":"ContainerDied","Data":"f1b44fc6d47ff9c9a7198ed33bbe077e8e13b075b32e0afddd26a09be6374eb1"} Oct 30 18:44:38.417253 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:38.417177 4455 scope.go:122] "RemoveContainer" containerID="6876f31cc94d72383d5346128cd01f57e4cfd518824d39c00be506c9c15a4bc8" Oct 30 18:44:38.418848 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:38.418822 4455 generic.go:356] "Generic (PLEG): container finished" podID="1e050532-86d8-4954-ae7a-b0fcd484ad24" containerID="eeda27f88a2295a5daf43a0093fc91d5b02d804fd8aec3821f95168389ec5491" exitCode=0 Oct 30 18:44:38.419140 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:38.418978 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b424a5-7d97cc8c87-vvjcw" event={"ID":"1e050532-86d8-4954-ae7a-b0fcd484ad24","Type":"ContainerDied","Data":"eeda27f88a2295a5daf43a0093fc91d5b02d804fd8aec3821f95168389ec5491"} Oct 30 18:44:38.419140 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:38.419022 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b424a5-7d97cc8c87-vvjcw" event={"ID":"1e050532-86d8-4954-ae7a-b0fcd484ad24","Type":"ContainerDied","Data":"8459fc63bd7e472105839c9984b8e3ef873cec5df7bbe3fc2cca5ee89035399e"} Oct 30 18:44:38.419140 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:38.419102 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b424a5-7d97cc8c87-vvjcw" Oct 30 18:44:38.429587 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:38.429569 4455 scope.go:122] "RemoveContainer" containerID="6876f31cc94d72383d5346128cd01f57e4cfd518824d39c00be506c9c15a4bc8" Oct 30 18:44:38.429851 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:44:38.429834 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6876f31cc94d72383d5346128cd01f57e4cfd518824d39c00be506c9c15a4bc8\": container with ID starting with 6876f31cc94d72383d5346128cd01f57e4cfd518824d39c00be506c9c15a4bc8 not found: ID does not exist" containerID="6876f31cc94d72383d5346128cd01f57e4cfd518824d39c00be506c9c15a4bc8" Oct 30 18:44:38.429901 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:38.429859 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6876f31cc94d72383d5346128cd01f57e4cfd518824d39c00be506c9c15a4bc8"} err="failed to get container status \"6876f31cc94d72383d5346128cd01f57e4cfd518824d39c00be506c9c15a4bc8\": rpc error: code = NotFound desc = could not find container \"6876f31cc94d72383d5346128cd01f57e4cfd518824d39c00be506c9c15a4bc8\": container with ID starting with 6876f31cc94d72383d5346128cd01f57e4cfd518824d39c00be506c9c15a4bc8 not found: ID does not exist" Oct 30 18:44:38.429901 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:38.429875 4455 scope.go:122] "RemoveContainer" containerID="eeda27f88a2295a5daf43a0093fc91d5b02d804fd8aec3821f95168389ec5491" Oct 30 18:44:38.439503 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:38.439479 4455 scope.go:122] "RemoveContainer" containerID="eeda27f88a2295a5daf43a0093fc91d5b02d804fd8aec3821f95168389ec5491" Oct 30 18:44:38.439789 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:44:38.439770 4455 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeda27f88a2295a5daf43a0093fc91d5b02d804fd8aec3821f95168389ec5491\": container with ID starting with eeda27f88a2295a5daf43a0093fc91d5b02d804fd8aec3821f95168389ec5491 not found: ID does not exist" containerID="eeda27f88a2295a5daf43a0093fc91d5b02d804fd8aec3821f95168389ec5491" Oct 30 18:44:38.439829 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:38.439798 4455 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeda27f88a2295a5daf43a0093fc91d5b02d804fd8aec3821f95168389ec5491"} err="failed to get container status \"eeda27f88a2295a5daf43a0093fc91d5b02d804fd8aec3821f95168389ec5491\": rpc error: code = NotFound desc = could not find container \"eeda27f88a2295a5daf43a0093fc91d5b02d804fd8aec3821f95168389ec5491\": container with ID starting with eeda27f88a2295a5daf43a0093fc91d5b02d804fd8aec3821f95168389ec5491 not found: ID does not exist" Oct 30 18:44:38.493593 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:38.493505 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b424a5-b698d4bb8-vvqmn"] Oct 30 18:44:38.495954 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:38.495934 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b424a5-b698d4bb8-vvqmn"] Oct 30 18:44:38.520991 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:38.520952 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b424a5-7d97cc8c87-vvjcw"] Oct 30 18:44:38.525219 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:38.525193 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b424a5-7d97cc8c87-vvjcw"] Oct 30 18:44:39.056938 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:39.056887 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-75d56799cb-n8rw5_9eea1448-1cc7-4ff1-b584-2b7f3e1b730d/authorino/0.log" Oct 30 18:44:39.296575 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:39.296539 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-779bff956b-bmwkw_8e4d9f9b-bd72-4d8b-a533-79334334789f/manager/0.log" Oct 30 18:44:39.634287 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:39.634256 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e050532-86d8-4954-ae7a-b0fcd484ad24" path="/var/lib/kubelet/pods/1e050532-86d8-4954-ae7a-b0fcd484ad24/volumes" Oct 30 18:44:39.634728 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:39.634705 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61033891-2264-449f-9a4b-d5b775d51ca6" path="/var/lib/kubelet/pods/61033891-2264-449f-9a4b-d5b775d51ca6/volumes" Oct 30 18:44:42.566701 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:42.566657 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:44:42.567277 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:42.566721 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" podUID="c7c21be6-b9eb-401c-aa3c-af5b62960cd8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:44:42.942767 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:42.942727 4455 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 18:44:42.943029 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:42.942973 4455 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" podUID="4e7b0c73-0041-4549-b47e-76a7aaed2b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 18:44:43.452285 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.452244 4455 generic.go:356] "Generic (PLEG): container finished" podID="4e7b0c73-0041-4549-b47e-76a7aaed2b56" containerID="79d17f879566861647cf69447a10eeff41d3d2ff0ccd53c84938d1107672ab0f" exitCode=0 Oct 30 18:44:43.452285 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.452409 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" event={"ID":"4e7b0c73-0041-4549-b47e-76a7aaed2b56","Type":"ContainerDied","Data":"79d17f879566861647cf69447a10eeff41d3d2ff0ccd53c84938d1107672ab0f"} Oct 30 18:44:43.455416 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.454582 4455 generic.go:356] "Generic (PLEG): container finished" podID="c7c21be6-b9eb-401c-aa3c-af5b62960cd8" containerID="2e57522cd1c5af27c444736562b4275230f8fe90ef6a33f7e7ef157c5cc09714" exitCode=0 Oct 30 18:44:43.455416 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.454669 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" event={"ID":"c7c21be6-b9eb-401c-aa3c-af5b62960cd8","Type":"ContainerDied","Data":"2e57522cd1c5af27c444736562b4275230f8fe90ef6a33f7e7ef157c5cc09714"} Oct 30 18:44:43.456600 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.456548 4455 generic.go:356] "Generic (PLEG): container finished" podID="1d3471dc-aaf3-4060-a64d-2413e522e65a" containerID="af56ed4e6bb5f101fe282f280b2506a9145a50eefe1f248c8a104b25b4ae0d67" exitCode=0 Oct 30 18:44:43.456711 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.456625 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" event={"ID":"1d3471dc-aaf3-4060-a64d-2413e522e65a","Type":"ContainerDied","Data":"af56ed4e6bb5f101fe282f280b2506a9145a50eefe1f248c8a104b25b4ae0d67"} Oct 30 18:44:43.600298 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.600268 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" Oct 30 18:44:43.643338 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.643297 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" Oct 30 18:44:43.646791 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.646769 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" Oct 30 18:44:43.684153 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.684066 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d3471dc-aaf3-4060-a64d-2413e522e65a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d3471dc-aaf3-4060-a64d-2413e522e65a-workload-socket\") pod \"1d3471dc-aaf3-4060-a64d-2413e522e65a\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " Oct 30 18:44:43.684347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.684158 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d3471dc-aaf3-4060-a64d-2413e522e65a-istio-token\" (UniqueName: \"kubernetes.io/projected/1d3471dc-aaf3-4060-a64d-2413e522e65a-istio-token\") pod \"1d3471dc-aaf3-4060-a64d-2413e522e65a\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " Oct 30 18:44:43.684347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.684188 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d3471dc-aaf3-4060-a64d-2413e522e65a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d3471dc-aaf3-4060-a64d-2413e522e65a-credential-socket\") pod \"1d3471dc-aaf3-4060-a64d-2413e522e65a\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " Oct 30 18:44:43.684347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.684246 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d3471dc-aaf3-4060-a64d-2413e522e65a-kube-api-access-5r4ff\" (UniqueName: \"kubernetes.io/projected/1d3471dc-aaf3-4060-a64d-2413e522e65a-kube-api-access-5r4ff\") pod \"1d3471dc-aaf3-4060-a64d-2413e522e65a\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " Oct 30 18:44:43.684347 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.684301 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d3471dc-aaf3-4060-a64d-2413e522e65a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d3471dc-aaf3-4060-a64d-2413e522e65a-workload-certs\") pod \"1d3471dc-aaf3-4060-a64d-2413e522e65a\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " Oct 30 18:44:43.684596 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.684353 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d3471dc-aaf3-4060-a64d-2413e522e65a-workload-socket" pod "1d3471dc-aaf3-4060-a64d-2413e522e65a" (UID: "1d3471dc-aaf3-4060-a64d-2413e522e65a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.684596 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.684434 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d3471dc-aaf3-4060-a64d-2413e522e65a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d3471dc-aaf3-4060-a64d-2413e522e65a-istio-podinfo\") pod \"1d3471dc-aaf3-4060-a64d-2413e522e65a\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " Oct 30 18:44:43.684596 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.684480 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d3471dc-aaf3-4060-a64d-2413e522e65a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d3471dc-aaf3-4060-a64d-2413e522e65a-istio-data\") pod \"1d3471dc-aaf3-4060-a64d-2413e522e65a\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " Oct 30 18:44:43.684596 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.684519 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d3471dc-aaf3-4060-a64d-2413e522e65a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d3471dc-aaf3-4060-a64d-2413e522e65a-istio-envoy\") pod \"1d3471dc-aaf3-4060-a64d-2413e522e65a\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " Oct 30 18:44:43.684596 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.684563 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d3471dc-aaf3-4060-a64d-2413e522e65a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d3471dc-aaf3-4060-a64d-2413e522e65a-istiod-ca-cert\") pod \"1d3471dc-aaf3-4060-a64d-2413e522e65a\" (UID: \"1d3471dc-aaf3-4060-a64d-2413e522e65a\") " Oct 30 18:44:43.684873 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.684603 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d3471dc-aaf3-4060-a64d-2413e522e65a-credential-socket" pod "1d3471dc-aaf3-4060-a64d-2413e522e65a" (UID: "1d3471dc-aaf3-4060-a64d-2413e522e65a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.684873 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.684708 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d3471dc-aaf3-4060-a64d-2413e522e65a-workload-certs" pod "1d3471dc-aaf3-4060-a64d-2413e522e65a" (UID: "1d3471dc-aaf3-4060-a64d-2413e522e65a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.685159 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.684874 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d3471dc-aaf3-4060-a64d-2413e522e65a-istio-data" pod "1d3471dc-aaf3-4060-a64d-2413e522e65a" (UID: "1d3471dc-aaf3-4060-a64d-2413e522e65a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.685159 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.685045 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d3471dc-aaf3-4060-a64d-2413e522e65a-istiod-ca-cert" pod "1d3471dc-aaf3-4060-a64d-2413e522e65a" (UID: "1d3471dc-aaf3-4060-a64d-2413e522e65a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:43.685159 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.685114 4455 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d3471dc-aaf3-4060-a64d-2413e522e65a-workload-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.685159 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.685136 4455 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d3471dc-aaf3-4060-a64d-2413e522e65a-istio-data\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.685412 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.685165 4455 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d3471dc-aaf3-4060-a64d-2413e522e65a-workload-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.685412 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.685179 4455 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d3471dc-aaf3-4060-a64d-2413e522e65a-credential-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.687235 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.687201 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d3471dc-aaf3-4060-a64d-2413e522e65a-istio-token" pod "1d3471dc-aaf3-4060-a64d-2413e522e65a" (UID: "1d3471dc-aaf3-4060-a64d-2413e522e65a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:43.687581 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.687559 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d3471dc-aaf3-4060-a64d-2413e522e65a-istio-envoy" pod "1d3471dc-aaf3-4060-a64d-2413e522e65a" (UID: "1d3471dc-aaf3-4060-a64d-2413e522e65a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.687683 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.687582 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d3471dc-aaf3-4060-a64d-2413e522e65a-kube-api-access-5r4ff" pod "1d3471dc-aaf3-4060-a64d-2413e522e65a" (UID: "1d3471dc-aaf3-4060-a64d-2413e522e65a"). InnerVolumeSpecName "kube-api-access-5r4ff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:43.687762 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.687674 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d3471dc-aaf3-4060-a64d-2413e522e65a-istio-podinfo" pod "1d3471dc-aaf3-4060-a64d-2413e522e65a" (UID: "1d3471dc-aaf3-4060-a64d-2413e522e65a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:44:43.786379 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.786340 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e7b0c73-0041-4549-b47e-76a7aaed2b56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7b0c73-0041-4549-b47e-76a7aaed2b56-istiod-ca-cert\") pod \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " Oct 30 18:44:43.786592 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.786441 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e7b0c73-0041-4549-b47e-76a7aaed2b56-istio-token\" (UniqueName: \"kubernetes.io/projected/4e7b0c73-0041-4549-b47e-76a7aaed2b56-istio-token\") pod \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " Oct 30 18:44:43.786592 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.786478 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7b0c73-0041-4549-b47e-76a7aaed2b56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7b0c73-0041-4549-b47e-76a7aaed2b56-istio-envoy\") pod \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " Oct 30 18:44:43.786592 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.786521 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-istiod-ca-cert\") pod \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " Oct 30 18:44:43.786592 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.786574 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7b0c73-0041-4549-b47e-76a7aaed2b56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7b0c73-0041-4549-b47e-76a7aaed2b56-credential-socket\") pod \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " Oct 30 18:44:43.786779 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.786615 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-istio-envoy\") pod \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " Oct 30 18:44:43.786779 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.786643 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7b0c73-0041-4549-b47e-76a7aaed2b56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7b0c73-0041-4549-b47e-76a7aaed2b56-workload-socket\") pod \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " Oct 30 18:44:43.786779 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.786700 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-istio-data\") pod \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " Oct 30 18:44:43.786779 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.786704 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e7b0c73-0041-4549-b47e-76a7aaed2b56-istiod-ca-cert" pod "4e7b0c73-0041-4549-b47e-76a7aaed2b56" (UID: "4e7b0c73-0041-4549-b47e-76a7aaed2b56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:43.786779 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.786727 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-workload-certs\") pod \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " Oct 30 18:44:43.786779 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.786750 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-istio-podinfo\") pod \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " Oct 30 18:44:43.786779 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.786770 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e7b0c73-0041-4549-b47e-76a7aaed2b56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7b0c73-0041-4549-b47e-76a7aaed2b56-istio-podinfo\") pod \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " Oct 30 18:44:43.787167 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.786792 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-istio-token\" (UniqueName: \"kubernetes.io/projected/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-istio-token\") pod \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " Oct 30 18:44:43.787167 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.786808 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7b0c73-0041-4549-b47e-76a7aaed2b56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7b0c73-0041-4549-b47e-76a7aaed2b56-istio-data\") pod \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " Oct 30 18:44:43.787167 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.786845 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-workload-socket\") pod \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " Oct 30 18:44:43.787167 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.786864 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-credential-socket\") pod \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " Oct 30 18:44:43.787167 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.786933 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7b0c73-0041-4549-b47e-76a7aaed2b56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7b0c73-0041-4549-b47e-76a7aaed2b56-workload-certs\") pod \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " Oct 30 18:44:43.787167 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.786958 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e7b0c73-0041-4549-b47e-76a7aaed2b56-kube-api-access-9v685\" (UniqueName: \"kubernetes.io/projected/4e7b0c73-0041-4549-b47e-76a7aaed2b56-kube-api-access-9v685\") pod \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\" (UID: \"4e7b0c73-0041-4549-b47e-76a7aaed2b56\") " Oct 30 18:44:43.787167 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.787005 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-kube-api-access-g7rf7\" (UniqueName: \"kubernetes.io/projected/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-kube-api-access-g7rf7\") pod \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\" (UID: \"c7c21be6-b9eb-401c-aa3c-af5b62960cd8\") " Oct 30 18:44:43.787590 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.787164 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-istio-data" pod "c7c21be6-b9eb-401c-aa3c-af5b62960cd8" (UID: "c7c21be6-b9eb-401c-aa3c-af5b62960cd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.787590 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.787199 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-istiod-ca-cert" pod "c7c21be6-b9eb-401c-aa3c-af5b62960cd8" (UID: "c7c21be6-b9eb-401c-aa3c-af5b62960cd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:43.787590 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.787428 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7b0c73-0041-4549-b47e-76a7aaed2b56-workload-socket" pod "4e7b0c73-0041-4549-b47e-76a7aaed2b56" (UID: "4e7b0c73-0041-4549-b47e-76a7aaed2b56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.787771 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.787716 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7b0c73-0041-4549-b47e-76a7aaed2b56-istio-data" pod "4e7b0c73-0041-4549-b47e-76a7aaed2b56" (UID: "4e7b0c73-0041-4549-b47e-76a7aaed2b56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.787871 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.787845 4455 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d3471dc-aaf3-4060-a64d-2413e522e65a-istio-podinfo\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.788257 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.787890 4455 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d3471dc-aaf3-4060-a64d-2413e522e65a-istio-envoy\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.788257 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.787904 4455 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d3471dc-aaf3-4060-a64d-2413e522e65a-istiod-ca-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.788257 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.787918 4455 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7b0c73-0041-4549-b47e-76a7aaed2b56-istiod-ca-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.788257 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.787930 4455 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d3471dc-aaf3-4060-a64d-2413e522e65a-istio-token\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.788257 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.787934 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7b0c73-0041-4549-b47e-76a7aaed2b56-workload-certs" pod "4e7b0c73-0041-4549-b47e-76a7aaed2b56" (UID: "4e7b0c73-0041-4549-b47e-76a7aaed2b56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.788257 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.787943 4455 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-istiod-ca-cert\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.788257 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.787981 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r4ff\" (UniqueName: \"kubernetes.io/projected/1d3471dc-aaf3-4060-a64d-2413e522e65a-kube-api-access-5r4ff\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.788257 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.788000 4455 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7b0c73-0041-4549-b47e-76a7aaed2b56-workload-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.788257 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.788017 4455 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-istio-data\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.788257 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.788030 4455 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7b0c73-0041-4549-b47e-76a7aaed2b56-istio-data\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.790394 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.789587 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7b0c73-0041-4549-b47e-76a7aaed2b56-istio-envoy" pod "4e7b0c73-0041-4549-b47e-76a7aaed2b56" (UID: "4e7b0c73-0041-4549-b47e-76a7aaed2b56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.790394 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.789633 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-istio-podinfo" pod "c7c21be6-b9eb-401c-aa3c-af5b62960cd8" (UID: "c7c21be6-b9eb-401c-aa3c-af5b62960cd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:44:43.790394 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.789886 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-workload-certs" pod "c7c21be6-b9eb-401c-aa3c-af5b62960cd8" (UID: "c7c21be6-b9eb-401c-aa3c-af5b62960cd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.790394 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.789913 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7b0c73-0041-4549-b47e-76a7aaed2b56-credential-socket" pod "4e7b0c73-0041-4549-b47e-76a7aaed2b56" (UID: "4e7b0c73-0041-4549-b47e-76a7aaed2b56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.790394 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.790192 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e7b0c73-0041-4549-b47e-76a7aaed2b56-istio-podinfo" pod "4e7b0c73-0041-4549-b47e-76a7aaed2b56" (UID: "4e7b0c73-0041-4549-b47e-76a7aaed2b56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:44:43.790394 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.790212 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-istio-token" pod "c7c21be6-b9eb-401c-aa3c-af5b62960cd8" (UID: "c7c21be6-b9eb-401c-aa3c-af5b62960cd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:43.790394 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.790312 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-credential-socket" pod "c7c21be6-b9eb-401c-aa3c-af5b62960cd8" (UID: "c7c21be6-b9eb-401c-aa3c-af5b62960cd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.790934 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.790521 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-istio-envoy" pod "c7c21be6-b9eb-401c-aa3c-af5b62960cd8" (UID: "c7c21be6-b9eb-401c-aa3c-af5b62960cd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.791002 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.789700 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-workload-socket" pod "c7c21be6-b9eb-401c-aa3c-af5b62960cd8" (UID: "c7c21be6-b9eb-401c-aa3c-af5b62960cd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.792345 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.792285 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e7b0c73-0041-4549-b47e-76a7aaed2b56-kube-api-access-9v685" pod "4e7b0c73-0041-4549-b47e-76a7aaed2b56" (UID: "4e7b0c73-0041-4549-b47e-76a7aaed2b56"). InnerVolumeSpecName "kube-api-access-9v685". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:43.792345 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.792298 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e7b0c73-0041-4549-b47e-76a7aaed2b56-istio-token" pod "4e7b0c73-0041-4549-b47e-76a7aaed2b56" (UID: "4e7b0c73-0041-4549-b47e-76a7aaed2b56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:43.792649 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.792626 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-kube-api-access-g7rf7" pod "c7c21be6-b9eb-401c-aa3c-af5b62960cd8" (UID: "c7c21be6-b9eb-401c-aa3c-af5b62960cd8"). InnerVolumeSpecName "kube-api-access-g7rf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:43.888940 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.888908 4455 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-workload-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.888940 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.888934 4455 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-credential-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.888940 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.888945 4455 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7b0c73-0041-4549-b47e-76a7aaed2b56-workload-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.889243 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.888954 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v685\" (UniqueName: \"kubernetes.io/projected/4e7b0c73-0041-4549-b47e-76a7aaed2b56-kube-api-access-9v685\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.889243 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.888964 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7rf7\" (UniqueName: \"kubernetes.io/projected/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-kube-api-access-g7rf7\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.889243 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.888973 4455 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e7b0c73-0041-4549-b47e-76a7aaed2b56-istio-token\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.889243 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.888981 4455 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7b0c73-0041-4549-b47e-76a7aaed2b56-istio-envoy\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.889243 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.888988 4455 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7b0c73-0041-4549-b47e-76a7aaed2b56-credential-socket\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.889243 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.888996 4455 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-istio-envoy\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.889243 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.889004 4455 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-workload-certs\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.889243 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.889011 4455 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-istio-podinfo\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.889243 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.889018 4455 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7b0c73-0041-4549-b47e-76a7aaed2b56-istio-podinfo\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.889243 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:43.889025 4455 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7c21be6-b9eb-401c-aa3c-af5b62960cd8-istio-token\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:44:44.462908 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:44.462881 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" Oct 30 18:44:44.463101 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:44.462876 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx" event={"ID":"4e7b0c73-0041-4549-b47e-76a7aaed2b56","Type":"ContainerDied","Data":"e90f0a27cd278d5d9dc55ef968521a7d31572d1883533571b84f071703d19156"} Oct 30 18:44:44.463101 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:44.463051 4455 scope.go:122] "RemoveContainer" containerID="79d17f879566861647cf69447a10eeff41d3d2ff0ccd53c84938d1107672ab0f" Oct 30 18:44:44.464580 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:44.464449 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" event={"ID":"c7c21be6-b9eb-401c-aa3c-af5b62960cd8","Type":"ContainerDied","Data":"23903a47680610f062477ecc91016d0254f146879b0de2dbdab9dd1322580e2b"} Oct 30 18:44:44.464580 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:44.464460 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx" Oct 30 18:44:44.466045 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:44.466023 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" Oct 30 18:44:44.466189 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:44.466024 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd" event={"ID":"1d3471dc-aaf3-4060-a64d-2413e522e65a","Type":"ContainerDied","Data":"3ea70c37d22d111a79b3374f1cc119caefe01505d709500a8cf0f74869fa5c0e"} Oct 30 18:44:44.478521 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:44.478479 4455 scope.go:122] "RemoveContainer" containerID="2e57522cd1c5af27c444736562b4275230f8fe90ef6a33f7e7ef157c5cc09714" Oct 30 18:44:44.487647 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:44.487627 4455 scope.go:122] "RemoveContainer" containerID="af56ed4e6bb5f101fe282f280b2506a9145a50eefe1f248c8a104b25b4ae0d67" Oct 30 18:44:44.526990 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:44.526959 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd"] Oct 30 18:44:44.531465 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:44.531440 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-5z9pd"] Oct 30 18:44:44.564854 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:44.564822 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx"] Oct 30 18:44:44.571846 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:44.571818 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-v5nqx"] Oct 30 18:44:44.609349 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:44.608691 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx"] Oct 30 18:44:44.618246 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:44.618208 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-kw5bx"] Oct 30 18:44:45.633473 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:45.633443 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d3471dc-aaf3-4060-a64d-2413e522e65a" path="/var/lib/kubelet/pods/1d3471dc-aaf3-4060-a64d-2413e522e65a/volumes" Oct 30 18:44:45.633920 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:45.633906 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e7b0c73-0041-4549-b47e-76a7aaed2b56" path="/var/lib/kubelet/pods/4e7b0c73-0041-4549-b47e-76a7aaed2b56/volumes" Oct 30 18:44:45.634303 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:45.634291 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7c21be6-b9eb-401c-aa3c-af5b62960cd8" path="/var/lib/kubelet/pods/c7c21be6-b9eb-401c-aa3c-af5b62960cd8/volumes" Oct 30 18:44:49.803982 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:49.803945 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-57f5c66455-l7hkg_64c72761-8ae7-47a1-aa47-d10cadeb3e34/manager/1.log" Oct 30 18:44:50.284312 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:50.284283 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6b575997cc-27c49_2cb9c001-fc99-4e94-908f-e3a59edeb3ab/manager/0.log" Oct 30 18:44:51.379891 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:51.379856 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-57f5c66455-l7hkg_64c72761-8ae7-47a1-aa47-d10cadeb3e34/manager/0.log" Oct 30 18:44:51.962589 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:51.962546 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-cf84bc4f4-dmpmz_7442c249-6e2c-4bcf-9383-b1ab7355e02e/manager/0.log" Oct 30 18:44:52.176337 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:52.176293 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xrlvs_0242f980-e7c2-4799-964d-b33944c19332/manager/0.log" Oct 30 18:44:52.394147 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:52.394119 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-5tcnq_0d007217-7dd2-4980-a42c-8e4df31e4293/manager/0.log" Oct 30 18:44:53.588407 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:53.588369 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-6kc8h_5e6cc7fd-dcda-490d-b3d0-661c28f95e1d/server/0.log" Oct 30 18:44:54.790470 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:44:54.790432 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-75d56799cb-n8rw5_9eea1448-1cc7-4ff1-b584-2b7f3e1b730d/authorino/0.log" Oct 30 18:45:00.151343 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.151286 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889765-vc77n"] Oct 30 18:45:00.151859 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.151842 4455 cpu_manager.go:414] "containerMap: removing container" podUID="dbdbb8b6-80ee-4abe-990e-72e4add7e77b" containerName="maas-api" Oct 30 18:45:00.151922 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.151861 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbdbb8b6-80ee-4abe-990e-72e4add7e77b" containerName="maas-api" Oct 30 18:45:00.151922 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.151879 4455 cpu_manager.go:414] "containerMap: removing container" podUID="15651f51-8abe-47fa-89f8-a9b30ee0b3e9" containerName="istio-proxy" Oct 30 18:45:00.151922 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.151888 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15651f51-8abe-47fa-89f8-a9b30ee0b3e9" containerName="istio-proxy" Oct 30 18:45:00.151922 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.151900 4455 cpu_manager.go:414] "containerMap: removing container" podUID="4e7b0c73-0041-4549-b47e-76a7aaed2b56" containerName="istio-proxy" Oct 30 18:45:00.151922 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.151904 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e7b0c73-0041-4549-b47e-76a7aaed2b56" containerName="istio-proxy" Oct 30 18:45:00.151922 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.151914 4455 cpu_manager.go:414] "containerMap: removing container" podUID="1524899f-95c7-4831-a90b-e108b715a3b3" containerName="maas-api" Oct 30 18:45:00.151922 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.151918 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1524899f-95c7-4831-a90b-e108b715a3b3" containerName="maas-api" Oct 30 18:45:00.151922 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.151925 4455 cpu_manager.go:414] "containerMap: removing container" podUID="96e42878-31cc-4d86-b428-e39a8aa72218" containerName="discovery" Oct 30 18:45:00.152173 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.151929 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96e42878-31cc-4d86-b428-e39a8aa72218" containerName="discovery" Oct 30 18:45:00.152173 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.151939 4455 cpu_manager.go:414] "containerMap: removing container" podUID="9d0733d2-85ad-4966-9740-97fffe7f54f6" containerName="maas-api" Oct 30 18:45:00.152173 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.151943 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d0733d2-85ad-4966-9740-97fffe7f54f6" containerName="maas-api" Oct 30 18:45:00.152173 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.151952 4455 cpu_manager.go:414] "containerMap: removing container" podUID="fc61777b-475d-4137-91eb-56e488ece861" containerName="payload-processing" Oct 30 18:45:00.152173 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.151956 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc61777b-475d-4137-91eb-56e488ece861" containerName="payload-processing" Oct 30 18:45:00.152173 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.151967 4455 cpu_manager.go:414] "containerMap: removing container" podUID="e62b028a-4ecb-46f8-9d6c-94bf236dc782" containerName="maas-api" Oct 30 18:45:00.152173 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.151971 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e62b028a-4ecb-46f8-9d6c-94bf236dc782" containerName="maas-api" Oct 30 18:45:00.152173 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.151984 4455 cpu_manager.go:414] "containerMap: removing container" podUID="1e050532-86d8-4954-ae7a-b0fcd484ad24" containerName="payload-pre-processing" Oct 30 18:45:00.152173 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.151988 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e050532-86d8-4954-ae7a-b0fcd484ad24" containerName="payload-pre-processing" Oct 30 18:45:00.152173 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.151996 4455 cpu_manager.go:414] "containerMap: removing container" podUID="c7c21be6-b9eb-401c-aa3c-af5b62960cd8" containerName="istio-proxy" Oct 30 18:45:00.152173 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.152001 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7c21be6-b9eb-401c-aa3c-af5b62960cd8" containerName="istio-proxy" Oct 30 18:45:00.152173 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.152009 4455 cpu_manager.go:414] "containerMap: removing container" podUID="61033891-2264-449f-9a4b-d5b775d51ca6" containerName="payload-processing" Oct 30 18:45:00.152173 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.152013 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61033891-2264-449f-9a4b-d5b775d51ca6" containerName="payload-processing" Oct 30 18:45:00.152173 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.152022 4455 cpu_manager.go:414] "containerMap: removing container" podUID="877c2eae-69e9-42fe-bbb9-b7a8ced3fa06" containerName="payload-pre-processing" Oct 30 18:45:00.152173 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.152026 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="877c2eae-69e9-42fe-bbb9-b7a8ced3fa06" containerName="payload-pre-processing" Oct 30 18:45:00.152173 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.152035 4455 cpu_manager.go:414] "containerMap: removing container" podUID="1d3471dc-aaf3-4060-a64d-2413e522e65a" containerName="istio-proxy" Oct 30 18:45:00.152173 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.152039 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d3471dc-aaf3-4060-a64d-2413e522e65a" containerName="istio-proxy" Oct 30 18:45:00.152173 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.152102 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="877c2eae-69e9-42fe-bbb9-b7a8ced3fa06" containerName="payload-pre-processing" Oct 30 18:45:00.152173 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.152110 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d3471dc-aaf3-4060-a64d-2413e522e65a" containerName="istio-proxy" Oct 30 18:45:00.152173 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.152117 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbdbb8b6-80ee-4abe-990e-72e4add7e77b" containerName="maas-api" Oct 30 18:45:00.152173 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.152125 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="15651f51-8abe-47fa-89f8-a9b30ee0b3e9" containerName="istio-proxy" Oct 30 18:45:00.152173 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.152131 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e7b0c73-0041-4549-b47e-76a7aaed2b56" containerName="istio-proxy" Oct 30 18:45:00.152173 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.152137 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="1524899f-95c7-4831-a90b-e108b715a3b3" containerName="maas-api" Oct 30 18:45:00.152173 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.152142 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="96e42878-31cc-4d86-b428-e39a8aa72218" containerName="discovery" Oct 30 18:45:00.152173 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.152148 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d0733d2-85ad-4966-9740-97fffe7f54f6" containerName="maas-api" Oct 30 18:45:00.152173 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.152153 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc61777b-475d-4137-91eb-56e488ece861" containerName="payload-processing" Oct 30 18:45:00.152173 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.152164 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e050532-86d8-4954-ae7a-b0fcd484ad24" containerName="payload-pre-processing" Oct 30 18:45:00.152173 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.152168 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7c21be6-b9eb-401c-aa3c-af5b62960cd8" containerName="istio-proxy" Oct 30 18:45:00.152173 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.152174 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="61033891-2264-449f-9a4b-d5b775d51ca6" containerName="payload-processing" Oct 30 18:45:00.155300 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.155284 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-vc77n" Oct 30 18:45:00.169566 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.169547 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:45:00.183748 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.183720 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889765-vc77n"] Oct 30 18:45:00.260017 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.259955 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8j8w\" (UniqueName: \"kubernetes.io/projected/88fa033c-88cc-4958-b5d0-3e5846d6f8a6-kube-api-access-g8j8w\") pod \"maas-api-key-cleanup-29889765-vc77n\" (UID: \"88fa033c-88cc-4958-b5d0-3e5846d6f8a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-vc77n" Oct 30 18:45:00.260238 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.260071 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88fa033c-88cc-4958-b5d0-3e5846d6f8a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889765-vc77n\" (UID: \"88fa033c-88cc-4958-b5d0-3e5846d6f8a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-vc77n" Oct 30 18:45:00.360823 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.360788 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8j8w\" (UniqueName: \"kubernetes.io/projected/88fa033c-88cc-4958-b5d0-3e5846d6f8a6-kube-api-access-g8j8w\") pod \"maas-api-key-cleanup-29889765-vc77n\" (UID: \"88fa033c-88cc-4958-b5d0-3e5846d6f8a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-vc77n" Oct 30 18:45:00.360997 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.360870 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88fa033c-88cc-4958-b5d0-3e5846d6f8a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889765-vc77n\" (UID: \"88fa033c-88cc-4958-b5d0-3e5846d6f8a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-vc77n" Oct 30 18:45:00.361573 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.361545 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88fa033c-88cc-4958-b5d0-3e5846d6f8a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889765-vc77n\" (UID: \"88fa033c-88cc-4958-b5d0-3e5846d6f8a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-vc77n" Oct 30 18:45:00.384621 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.384590 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8j8w\" (UniqueName: \"kubernetes.io/projected/88fa033c-88cc-4958-b5d0-3e5846d6f8a6-kube-api-access-g8j8w\") pod \"maas-api-key-cleanup-29889765-vc77n\" (UID: \"88fa033c-88cc-4958-b5d0-3e5846d6f8a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-vc77n" Oct 30 18:45:00.386343 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:45:00.386301 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:45:00.468207 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.468179 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-vc77n" Oct 30 18:45:00.771023 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:00.770952 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889765-vc77n"] Oct 30 18:45:00.774489 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:45:00.774446 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88fa033c_88cc_4958_b5d0_3e5846d6f8a6.slice/crio-07655e1c37de3bc4a1419a5b400cf9bf73a4dabe4df7e56b2484b48806bd39df WatchSource:0}: Error finding container 07655e1c37de3bc4a1419a5b400cf9bf73a4dabe4df7e56b2484b48806bd39df: Status 404 returned error can't find the container with id 07655e1c37de3bc4a1419a5b400cf9bf73a4dabe4df7e56b2484b48806bd39df Oct 30 18:45:01.568607 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:01.568565 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-vc77n" event={"ID":"88fa033c-88cc-4958-b5d0-3e5846d6f8a6","Type":"ContainerStarted","Data":"4b1a9688dd87c88f3fea397f0030a2e3abbbbc053e376bf3c921ac78ffbcf404"} Oct 30 18:45:01.569075 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:01.568626 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-vc77n" event={"ID":"88fa033c-88cc-4958-b5d0-3e5846d6f8a6","Type":"ContainerStarted","Data":"07655e1c37de3bc4a1419a5b400cf9bf73a4dabe4df7e56b2484b48806bd39df"} Oct 30 18:45:01.607951 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:01.607899 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-vc77n" podStartSLOduration=1.607887394 podStartE2EDuration="1.607887394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:01.606480822 +0000 UTC m=+3154.094117857" watchObservedRunningTime="2026-10-30 18:45:01.607887394 +0000 UTC m=+3154.095524428" Oct 30 18:45:02.575279 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:02.575240 4455 generic.go:356] "Generic (PLEG): container finished" podID="88fa033c-88cc-4958-b5d0-3e5846d6f8a6" containerID="4b1a9688dd87c88f3fea397f0030a2e3abbbbc053e376bf3c921ac78ffbcf404" exitCode=60 Oct 30 18:45:02.575702 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:02.575343 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-vc77n" event={"ID":"88fa033c-88cc-4958-b5d0-3e5846d6f8a6","Type":"ContainerDied","Data":"4b1a9688dd87c88f3fea397f0030a2e3abbbbc053e376bf3c921ac78ffbcf404"} Oct 30 18:45:02.575702 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:02.575624 4455 scope.go:122] "RemoveContainer" containerID="4b1a9688dd87c88f3fea397f0030a2e3abbbbc053e376bf3c921ac78ffbcf404" Oct 30 18:45:03.583614 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:03.583575 4455 generic.go:356] "Generic (PLEG): container finished" podID="88fa033c-88cc-4958-b5d0-3e5846d6f8a6" containerID="a2c93b2c9925649ad3060530186bbb6872c61f31cc858f165c1b739e0161a03b" exitCode=60 Oct 30 18:45:03.584115 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:03.583668 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-vc77n" event={"ID":"88fa033c-88cc-4958-b5d0-3e5846d6f8a6","Type":"ContainerDied","Data":"a2c93b2c9925649ad3060530186bbb6872c61f31cc858f165c1b739e0161a03b"} Oct 30 18:45:03.584115 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:03.583726 4455 scope.go:122] "RemoveContainer" containerID="4b1a9688dd87c88f3fea397f0030a2e3abbbbc053e376bf3c921ac78ffbcf404" Oct 30 18:45:03.584115 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:03.583988 4455 scope.go:122] "RemoveContainer" containerID="a2c93b2c9925649ad3060530186bbb6872c61f31cc858f165c1b739e0161a03b" Oct 30 18:45:03.584339 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:45:03.584287 4455 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889765-vc77n_odh-ai-gateway-infra(88fa033c-88cc-4958-b5d0-3e5846d6f8a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-vc77n" podUID="88fa033c-88cc-4958-b5d0-3e5846d6f8a6" Oct 30 18:45:04.590593 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:04.590566 4455 scope.go:122] "RemoveContainer" containerID="a2c93b2c9925649ad3060530186bbb6872c61f31cc858f165c1b739e0161a03b" Oct 30 18:45:04.590959 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:45:04.590819 4455 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889765-vc77n_odh-ai-gateway-infra(88fa033c-88cc-4958-b5d0-3e5846d6f8a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-vc77n" podUID="88fa033c-88cc-4958-b5d0-3e5846d6f8a6" Oct 30 18:45:15.626755 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:15.626719 4455 scope.go:122] "RemoveContainer" containerID="a2c93b2c9925649ad3060530186bbb6872c61f31cc858f165c1b739e0161a03b" Oct 30 18:45:16.658293 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:16.658260 4455 generic.go:356] "Generic (PLEG): container finished" podID="88fa033c-88cc-4958-b5d0-3e5846d6f8a6" containerID="100f6a46f516925b18c6d243863f3d20b0bae9e0dc23c63e836c095150147d31" exitCode=60 Oct 30 18:45:16.658750 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:16.658349 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-vc77n" event={"ID":"88fa033c-88cc-4958-b5d0-3e5846d6f8a6","Type":"ContainerDied","Data":"100f6a46f516925b18c6d243863f3d20b0bae9e0dc23c63e836c095150147d31"} Oct 30 18:45:16.658750 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:16.658407 4455 scope.go:122] "RemoveContainer" containerID="a2c93b2c9925649ad3060530186bbb6872c61f31cc858f165c1b739e0161a03b" Oct 30 18:45:16.658841 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:16.658776 4455 scope.go:122] "RemoveContainer" containerID="100f6a46f516925b18c6d243863f3d20b0bae9e0dc23c63e836c095150147d31" Oct 30 18:45:16.659042 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:45:16.659025 4455 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889765-vc77n_odh-ai-gateway-infra(88fa033c-88cc-4958-b5d0-3e5846d6f8a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-vc77n" podUID="88fa033c-88cc-4958-b5d0-3e5846d6f8a6" Oct 30 18:45:17.731540 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:17.731511 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889765-vc77n"] Oct 30 18:45:17.879127 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:17.879086 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-vc77n" Oct 30 18:45:17.932250 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:17.932209 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88fa033c-88cc-4958-b5d0-3e5846d6f8a6-kube-api-access-g8j8w\" (UniqueName: \"kubernetes.io/projected/88fa033c-88cc-4958-b5d0-3e5846d6f8a6-kube-api-access-g8j8w\") pod \"88fa033c-88cc-4958-b5d0-3e5846d6f8a6\" (UID: \"88fa033c-88cc-4958-b5d0-3e5846d6f8a6\") " Oct 30 18:45:17.932492 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:17.932354 4455 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88fa033c-88cc-4958-b5d0-3e5846d6f8a6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88fa033c-88cc-4958-b5d0-3e5846d6f8a6-maas-api-service-ca\") pod \"88fa033c-88cc-4958-b5d0-3e5846d6f8a6\" (UID: \"88fa033c-88cc-4958-b5d0-3e5846d6f8a6\") " Oct 30 18:45:17.933025 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:17.932994 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88fa033c-88cc-4958-b5d0-3e5846d6f8a6-maas-api-service-ca" pod "88fa033c-88cc-4958-b5d0-3e5846d6f8a6" (UID: "88fa033c-88cc-4958-b5d0-3e5846d6f8a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:17.934470 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:17.934440 4455 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88fa033c-88cc-4958-b5d0-3e5846d6f8a6-kube-api-access-g8j8w" pod "88fa033c-88cc-4958-b5d0-3e5846d6f8a6" (UID: "88fa033c-88cc-4958-b5d0-3e5846d6f8a6"). InnerVolumeSpecName "kube-api-access-g8j8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:18.034076 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:18.033981 4455 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8j8w\" (UniqueName: \"kubernetes.io/projected/88fa033c-88cc-4958-b5d0-3e5846d6f8a6-kube-api-access-g8j8w\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:45:18.034076 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:18.034025 4455 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88fa033c-88cc-4958-b5d0-3e5846d6f8a6-maas-api-service-ca\") on node \"ip-10-0-130-84.ec2.internal\" DevicePath \"\"" Oct 30 18:45:18.679239 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:18.679205 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-vc77n" event={"ID":"88fa033c-88cc-4958-b5d0-3e5846d6f8a6","Type":"ContainerDied","Data":"07655e1c37de3bc4a1419a5b400cf9bf73a4dabe4df7e56b2484b48806bd39df"} Oct 30 18:45:18.679239 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:18.679240 4455 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-vc77n" Oct 30 18:45:18.679517 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:18.679288 4455 scope.go:122] "RemoveContainer" containerID="100f6a46f516925b18c6d243863f3d20b0bae9e0dc23c63e836c095150147d31" Oct 30 18:45:18.729733 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:18.729698 4455 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889765-vc77n"] Oct 30 18:45:18.735617 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:18.735587 4455 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889765-vc77n"] Oct 30 18:45:19.632496 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:19.632463 4455 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88fa033c-88cc-4958-b5d0-3e5846d6f8a6" path="/var/lib/kubelet/pods/88fa033c-88cc-4958-b5d0-3e5846d6f8a6/volumes" Oct 30 18:45:30.384906 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:45:30.384867 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:45:45.396676 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:45.396587 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-cf84bc4f4-dmpmz_7442c249-6e2c-4bcf-9383-b1ab7355e02e/manager/0.log" Oct 30 18:45:45.618134 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:45.618100 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xrlvs_0242f980-e7c2-4799-964d-b33944c19332/manager/0.log" Oct 30 18:45:45.816100 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:45.816066 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-5tcnq_0d007217-7dd2-4980-a42c-8e4df31e4293/manager/0.log" Oct 30 18:45:46.813553 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:46.813526 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-6kc8h_5e6cc7fd-dcda-490d-b3d0-661c28f95e1d/server/0.log" Oct 30 18:45:48.829918 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:48.829880 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-fh9zp_725c3af8-4989-4b39-8141-5688451ea2bc/postgres/0.log" Oct 30 18:45:50.345580 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:45:50.345551 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-75d56799cb-n8rw5_9eea1448-1cc7-4ff1-b584-2b7f3e1b730d/authorino/0.log" Oct 30 18:46:00.384958 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:46:00.384917 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:46:00.588339 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:00.588277 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-779bff956b-bmwkw_8e4d9f9b-bd72-4d8b-a533-79334334789f/manager/0.log" Oct 30 18:46:01.045172 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:01.045139 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-57f5c66455-l7hkg_64c72761-8ae7-47a1-aa47-d10cadeb3e34/manager/1.log" Oct 30 18:46:01.585916 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:01.585815 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6b575997cc-27c49_2cb9c001-fc99-4e94-908f-e3a59edeb3ab/manager/0.log" Oct 30 18:46:02.644941 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:02.644911 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-57f5c66455-l7hkg_64c72761-8ae7-47a1-aa47-d10cadeb3e34/manager/0.log" Oct 30 18:46:03.167590 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:03.167552 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44_5ae819cc-7f64-4b4a-8559-1bc35afe67d1/istio-proxy/0.log" Oct 30 18:46:03.610887 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:03.610858 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-9zh44_c4fd1169-a9c3-4c6d-ba92-1a882c7309b0/discovery/0.log" Oct 30 18:46:04.206969 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:04.206936 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-nrbjr_d427f823-1810-42b1-b753-1790622bca7f/istio-proxy/0.log" Oct 30 18:46:04.620210 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:04.620177 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p_877f995b-cecd-4842-ba49-963b5c1f0590/istio-proxy/0.log" Oct 30 18:46:04.813557 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:04.813529 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-znczm_73b20a69-5907-4bec-a4c6-6b31b46231c4/payload-pre-processing/0.log" Oct 30 18:46:05.006225 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:05.006198 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-9l4x8_909b1ff6-6d4a-4887-9b59-efe2806ac0e8/payload-processing/0.log" Oct 30 18:46:06.011995 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:06.011962 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-rrpjm_e6e19981-c4a3-403e-9265-e33461b1d35a/main/0.log" Oct 30 18:46:06.434754 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:06.434669 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-nj9sw_efdc3c98-3ce3-4b85-90c1-27984b259887/main/0.log" Oct 30 18:46:06.838286 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:06.838255 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q2zzc_b7222c61-e786-4955-bd32-5f90a63ea48b/main/0.log" Oct 30 18:46:07.263651 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:07.263621 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-nphc9_484dd1d5-ee6a-4423-87c4-efbb4ba01767/main/0.log" Oct 30 18:46:30.384086 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:46:30.384046 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:46:34.193086 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:34.193051 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pxv85_1fb157ba-42c2-4efc-a76b-6863fc987fb6/global-pull-secret-syncer/0.log" Oct 30 18:46:34.881295 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:34.881267 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qzhll_73dbe36c-9ccb-47d6-b11e-afc35b87008e/konnectivity-agent/0.log" Oct 30 18:46:35.393405 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:35.393375 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-84.ec2.internal_da45dfd37aee5c345465266021945ad3/haproxy/0.log" Oct 30 18:46:41.786852 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:41.786819 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-75d56799cb-n8rw5_9eea1448-1cc7-4ff1-b584-2b7f3e1b730d/authorino/0.log" Oct 30 18:46:41.880885 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:41.880859 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-779bff956b-bmwkw_8e4d9f9b-bd72-4d8b-a533-79334334789f/manager/0.log" Oct 30 18:46:42.296157 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:42.296132 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-57f5c66455-l7hkg_64c72761-8ae7-47a1-aa47-d10cadeb3e34/manager/1.log" Oct 30 18:46:42.296362 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:42.296187 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-57f5c66455-l7hkg_64c72761-8ae7-47a1-aa47-d10cadeb3e34/manager/0.log" Oct 30 18:46:42.619839 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:42.619758 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6b575997cc-27c49_2cb9c001-fc99-4e94-908f-e3a59edeb3ab/manager/0.log" Oct 30 18:46:45.489565 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:45.489532 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d7fa1284-e214-4839-93d7-dd4b30ba4f34/alertmanager/0.log" Oct 30 18:46:45.514446 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:45.514417 4455 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lwz4v/perf-node-gather-daemonset-nfzwc"] Oct 30 18:46:45.514880 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:45.514865 4455 cpu_manager.go:414] "containerMap: removing container" podUID="88fa033c-88cc-4958-b5d0-3e5846d6f8a6" containerName="cleanup" Oct 30 18:46:45.514880 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:45.514879 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88fa033c-88cc-4958-b5d0-3e5846d6f8a6" containerName="cleanup" Oct 30 18:46:45.515023 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:45.514891 4455 cpu_manager.go:414] "containerMap: removing container" podUID="88fa033c-88cc-4958-b5d0-3e5846d6f8a6" containerName="cleanup" Oct 30 18:46:45.515023 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:45.514897 4455 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88fa033c-88cc-4958-b5d0-3e5846d6f8a6" containerName="cleanup" Oct 30 18:46:45.515023 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:45.514993 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="88fa033c-88cc-4958-b5d0-3e5846d6f8a6" containerName="cleanup" Oct 30 18:46:45.515023 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:45.515004 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="88fa033c-88cc-4958-b5d0-3e5846d6f8a6" containerName="cleanup" Oct 30 18:46:45.515023 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:45.515010 4455 memory_manager.go:365] "RemoveStaleState removing state" podUID="88fa033c-88cc-4958-b5d0-3e5846d6f8a6" containerName="cleanup" Oct 30 18:46:45.518481 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:45.518460 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-nfzwc" Oct 30 18:46:45.536762 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:45.536742 4455 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lwz4v\"/\"default-dockercfg-9zdsh\"" Oct 30 18:46:45.548381 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:45.548355 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lwz4v\"/\"kube-root-ca.crt\"" Oct 30 18:46:45.548559 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:45.548417 4455 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lwz4v\"/\"openshift-service-ca.crt\"" Oct 30 18:46:45.550880 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:45.550858 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lwz4v/perf-node-gather-daemonset-nfzwc"] Oct 30 18:46:45.602851 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:45.602822 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d7fa1284-e214-4839-93d7-dd4b30ba4f34/config-reloader/0.log" Oct 30 18:46:45.618261 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:45.618225 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cfa4d6af-d73f-40c5-82d7-399023b51530-lib-modules\") pod \"perf-node-gather-daemonset-nfzwc\" (UID: \"cfa4d6af-d73f-40c5-82d7-399023b51530\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-nfzwc" Oct 30 18:46:45.618416 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:45.618275 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cfa4d6af-d73f-40c5-82d7-399023b51530-podres\") pod \"perf-node-gather-daemonset-nfzwc\" (UID: \"cfa4d6af-d73f-40c5-82d7-399023b51530\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-nfzwc" Oct 30 18:46:45.618416 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:45.618382 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cfa4d6af-d73f-40c5-82d7-399023b51530-sys\") pod \"perf-node-gather-daemonset-nfzwc\" (UID: \"cfa4d6af-d73f-40c5-82d7-399023b51530\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-nfzwc" Oct 30 18:46:45.618416 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:45.618412 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cfa4d6af-d73f-40c5-82d7-399023b51530-proc\") pod \"perf-node-gather-daemonset-nfzwc\" (UID: \"cfa4d6af-d73f-40c5-82d7-399023b51530\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-nfzwc" Oct 30 18:46:45.618532 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:45.618507 4455 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thx2b\" (UniqueName: \"kubernetes.io/projected/cfa4d6af-d73f-40c5-82d7-399023b51530-kube-api-access-thx2b\") pod \"perf-node-gather-daemonset-nfzwc\" (UID: \"cfa4d6af-d73f-40c5-82d7-399023b51530\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-nfzwc" Oct 30 18:46:45.718989 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:45.718953 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cfa4d6af-d73f-40c5-82d7-399023b51530-podres\") pod \"perf-node-gather-daemonset-nfzwc\" (UID: \"cfa4d6af-d73f-40c5-82d7-399023b51530\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-nfzwc" Oct 30 18:46:45.719207 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:45.719053 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cfa4d6af-d73f-40c5-82d7-399023b51530-podres\") pod \"perf-node-gather-daemonset-nfzwc\" (UID: \"cfa4d6af-d73f-40c5-82d7-399023b51530\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-nfzwc" Oct 30 18:46:45.719207 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:45.719090 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cfa4d6af-d73f-40c5-82d7-399023b51530-sys\") pod \"perf-node-gather-daemonset-nfzwc\" (UID: \"cfa4d6af-d73f-40c5-82d7-399023b51530\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-nfzwc" Oct 30 18:46:45.719207 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:45.719133 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cfa4d6af-d73f-40c5-82d7-399023b51530-proc\") pod \"perf-node-gather-daemonset-nfzwc\" (UID: \"cfa4d6af-d73f-40c5-82d7-399023b51530\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-nfzwc" Oct 30 18:46:45.719207 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:45.719192 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cfa4d6af-d73f-40c5-82d7-399023b51530-sys\") pod \"perf-node-gather-daemonset-nfzwc\" (UID: \"cfa4d6af-d73f-40c5-82d7-399023b51530\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-nfzwc" Oct 30 18:46:45.719494 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:45.719200 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thx2b\" (UniqueName: \"kubernetes.io/projected/cfa4d6af-d73f-40c5-82d7-399023b51530-kube-api-access-thx2b\") pod \"perf-node-gather-daemonset-nfzwc\" (UID: \"cfa4d6af-d73f-40c5-82d7-399023b51530\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-nfzwc" Oct 30 18:46:45.719494 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:45.719266 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cfa4d6af-d73f-40c5-82d7-399023b51530-proc\") pod \"perf-node-gather-daemonset-nfzwc\" (UID: \"cfa4d6af-d73f-40c5-82d7-399023b51530\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-nfzwc" Oct 30 18:46:45.719494 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:45.719355 4455 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cfa4d6af-d73f-40c5-82d7-399023b51530-lib-modules\") pod \"perf-node-gather-daemonset-nfzwc\" (UID: \"cfa4d6af-d73f-40c5-82d7-399023b51530\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-nfzwc" Oct 30 18:46:45.719494 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:45.719465 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cfa4d6af-d73f-40c5-82d7-399023b51530-lib-modules\") pod \"perf-node-gather-daemonset-nfzwc\" (UID: \"cfa4d6af-d73f-40c5-82d7-399023b51530\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-nfzwc" Oct 30 18:46:45.722549 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:45.722529 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d7fa1284-e214-4839-93d7-dd4b30ba4f34/kube-rbac-proxy-web/0.log" Oct 30 18:46:45.741681 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:45.741617 4455 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thx2b\" (UniqueName: \"kubernetes.io/projected/cfa4d6af-d73f-40c5-82d7-399023b51530-kube-api-access-thx2b\") pod \"perf-node-gather-daemonset-nfzwc\" (UID: \"cfa4d6af-d73f-40c5-82d7-399023b51530\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-nfzwc" Oct 30 18:46:45.831211 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:45.831179 4455 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-nfzwc" Oct 30 18:46:45.832212 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:45.832182 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d7fa1284-e214-4839-93d7-dd4b30ba4f34/kube-rbac-proxy/0.log" Oct 30 18:46:45.947176 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:45.947150 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d7fa1284-e214-4839-93d7-dd4b30ba4f34/kube-rbac-proxy-metric/0.log" Oct 30 18:46:46.061111 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:46.061034 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d7fa1284-e214-4839-93d7-dd4b30ba4f34/prom-label-proxy/0.log" Oct 30 18:46:46.136354 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:46.136286 4455 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lwz4v/perf-node-gather-daemonset-nfzwc"] Oct 30 18:46:46.140217 ip-10-0-130-84 kubenswrapper[4455]: W1030 18:46:46.140191 4455 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcfa4d6af_d73f_40c5_82d7_399023b51530.slice/crio-859f907f3d780731e2b16a9c19265dd801acdd821dbfce86ec00b6637bde6c54 WatchSource:0}: Error finding container 859f907f3d780731e2b16a9c19265dd801acdd821dbfce86ec00b6637bde6c54: Status 404 returned error can't find the container with id 859f907f3d780731e2b16a9c19265dd801acdd821dbfce86ec00b6637bde6c54 Oct 30 18:46:46.154283 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:46.154260 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-nfzwc" event={"ID":"cfa4d6af-d73f-40c5-82d7-399023b51530","Type":"ContainerStarted","Data":"859f907f3d780731e2b16a9c19265dd801acdd821dbfce86ec00b6637bde6c54"} Oct 30 18:46:46.179405 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:46.179383 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d7fa1284-e214-4839-93d7-dd4b30ba4f34/init-config-reloader/0.log" Oct 30 18:46:46.427533 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:46.427410 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-m5pgt_75d3a246-8a03-458d-b744-c7910ba39c07/kube-state-metrics/0.log" Oct 30 18:46:46.552089 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:46.552061 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-m5pgt_75d3a246-8a03-458d-b744-c7910ba39c07/kube-rbac-proxy-main/0.log" Oct 30 18:46:46.676644 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:46.676616 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-m5pgt_75d3a246-8a03-458d-b744-c7910ba39c07/kube-rbac-proxy-self/0.log" Oct 30 18:46:46.810836 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:46.810808 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-cf84bc4f4-dmpmz_7442c249-6e2c-4bcf-9383-b1ab7355e02e/manager/0.log" Oct 30 18:46:46.867776 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:46.867731 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xrlvs_0242f980-e7c2-4799-964d-b33944c19332/manager/0.log" Oct 30 18:46:46.920221 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:46.920179 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-cf84bc4f4-dmpmz_7442c249-6e2c-4bcf-9383-b1ab7355e02e/manager/0.log" Oct 30 18:46:46.933010 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:46.932981 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-jd7b4_15a3b1ff-f48e-41d8-b4cb-90cd252d940b/monitoring-plugin/0.log" Oct 30 18:46:46.961888 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:46.961865 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-5tcnq_0d007217-7dd2-4980-a42c-8e4df31e4293/manager/0.log" Oct 30 18:46:46.979176 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:46.979121 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xrlvs_0242f980-e7c2-4799-964d-b33944c19332/manager/0.log" Oct 30 18:46:47.081093 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:47.081003 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-5tcnq_0d007217-7dd2-4980-a42c-8e4df31e4293/manager/0.log" Oct 30 18:46:47.160615 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:47.160579 4455 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-nfzwc" event={"ID":"cfa4d6af-d73f-40c5-82d7-399023b51530","Type":"ContainerStarted","Data":"f146bab6746547138685ababb7170cc41f3a9621f2cf72daaa06859651f59038"} Oct 30 18:46:47.160812 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:47.160735 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-nfzwc" Oct 30 18:46:47.208542 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:47.208484 4455 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-nfzwc" podStartSLOduration=2.2084676500000002 podStartE2EDuration="2.20846765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:47.206006713 +0000 UTC m=+3259.693643746" watchObservedRunningTime="2026-10-30 18:46:47.20846765 +0000 UTC m=+3259.696104685" Oct 30 18:46:47.818339 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:47.818290 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tflc2_d2a2fc35-0948-4648-b98a-086d173ed454/node-exporter/0.log" Oct 30 18:46:47.949636 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:47.949607 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tflc2_d2a2fc35-0948-4648-b98a-086d173ed454/kube-rbac-proxy/0.log" Oct 30 18:46:48.075341 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:48.075248 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tflc2_d2a2fc35-0948-4648-b98a-086d173ed454/init-textfile/0.log" Oct 30 18:46:48.290335 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:48.290299 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-hr4pf_1aac1dd8-54d7-4df2-b435-e14842f19975/kube-rbac-proxy-main/0.log" Oct 30 18:46:48.487204 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:48.487176 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-hr4pf_1aac1dd8-54d7-4df2-b435-e14842f19975/kube-rbac-proxy-self/0.log" Oct 30 18:46:48.623093 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:48.623061 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-hr4pf_1aac1dd8-54d7-4df2-b435-e14842f19975/openshift-state-metrics/0.log" Oct 30 18:46:48.835567 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:48.835496 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-6kc8h_5e6cc7fd-dcda-490d-b3d0-661c28f95e1d/server/0.log" Oct 30 18:46:48.955655 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:48.955626 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-6kc8h_5e6cc7fd-dcda-490d-b3d0-661c28f95e1d/server/0.log" Oct 30 18:46:50.130675 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:50.130643 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-wxt9d_33b2e16b-dd03-453f-94c4-0fb6cbed0701/prometheus-operator/0.log" Oct 30 18:46:50.239872 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:50.239848 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-wxt9d_33b2e16b-dd03-453f-94c4-0fb6cbed0701/kube-rbac-proxy/0.log" Oct 30 18:46:50.364521 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:50.364495 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-dh7v6_87ebbcab-1d74-4684-96d7-a61a85c39873/prometheus-operator-admission-webhook/0.log" Oct 30 18:46:50.858297 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:50.858267 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b65dfd95c-n9j9b_526f0988-8961-458a-b22b-8b0b983d1e0a/thanos-query/0.log" Oct 30 18:46:50.971276 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:50.971238 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b65dfd95c-n9j9b_526f0988-8961-458a-b22b-8b0b983d1e0a/kube-rbac-proxy-web/0.log" Oct 30 18:46:51.081736 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:51.081696 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b65dfd95c-n9j9b_526f0988-8961-458a-b22b-8b0b983d1e0a/kube-rbac-proxy/0.log" Oct 30 18:46:51.194266 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:51.194190 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b65dfd95c-n9j9b_526f0988-8961-458a-b22b-8b0b983d1e0a/prom-label-proxy/0.log" Oct 30 18:46:51.304451 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:51.304419 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b65dfd95c-n9j9b_526f0988-8961-458a-b22b-8b0b983d1e0a/kube-rbac-proxy-rules/0.log" Oct 30 18:46:51.312114 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:51.312075 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-cf84bc4f4-dmpmz_7442c249-6e2c-4bcf-9383-b1ab7355e02e/manager/0.log" Oct 30 18:46:51.362486 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:51.362463 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xrlvs_0242f980-e7c2-4799-964d-b33944c19332/manager/0.log" Oct 30 18:46:51.401167 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:51.401138 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-cf84bc4f4-dmpmz_7442c249-6e2c-4bcf-9383-b1ab7355e02e/manager/0.log" Oct 30 18:46:51.430607 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:51.430583 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b65dfd95c-n9j9b_526f0988-8961-458a-b22b-8b0b983d1e0a/kube-rbac-proxy-metrics/0.log" Oct 30 18:46:51.458667 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:51.458598 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-5tcnq_0d007217-7dd2-4980-a42c-8e4df31e4293/manager/0.log" Oct 30 18:46:51.472632 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:51.472611 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xrlvs_0242f980-e7c2-4799-964d-b33944c19332/manager/0.log" Oct 30 18:46:51.563299 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:51.563270 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-5tcnq_0d007217-7dd2-4980-a42c-8e4df31e4293/manager/0.log" Oct 30 18:46:53.177142 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:53.177115 4455 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-nfzwc" Oct 30 18:46:53.300066 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:53.300043 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-6kc8h_5e6cc7fd-dcda-490d-b3d0-661c28f95e1d/server/0.log" Oct 30 18:46:53.398259 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:53.398232 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-6kc8h_5e6cc7fd-dcda-490d-b3d0-661c28f95e1d/server/0.log" Oct 30 18:46:55.612428 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:55.612386 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-77dbb5d6f-5t9t6_4e182135-7cf7-4362-8f51-95b7b3173965/console/0.log" Oct 30 18:46:55.744017 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:55.743980 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-nbj4c_210dfba3-f3d3-48db-983d-112569110049/download-server/0.log" Oct 30 18:46:55.870866 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:55.870780 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-cf84bc4f4-dmpmz_7442c249-6e2c-4bcf-9383-b1ab7355e02e/manager/0.log" Oct 30 18:46:55.924469 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:55.924441 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xrlvs_0242f980-e7c2-4799-964d-b33944c19332/manager/0.log" Oct 30 18:46:56.011591 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:56.011561 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-5tcnq_0d007217-7dd2-4980-a42c-8e4df31e4293/manager/0.log" Oct 30 18:46:56.184913 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:56.184838 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-cf84bc4f4-dmpmz_7442c249-6e2c-4bcf-9383-b1ab7355e02e/manager/0.log" Oct 30 18:46:56.241217 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:56.241187 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xrlvs_0242f980-e7c2-4799-964d-b33944c19332/manager/0.log" Oct 30 18:46:56.338427 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:56.338401 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-5tcnq_0d007217-7dd2-4980-a42c-8e4df31e4293/manager/0.log" Oct 30 18:46:57.751735 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:57.751709 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-6kc8h_5e6cc7fd-dcda-490d-b3d0-661c28f95e1d/server/0.log" Oct 30 18:46:58.210344 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:58.210258 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-6kc8h_5e6cc7fd-dcda-490d-b3d0-661c28f95e1d/server/0.log" Oct 30 18:46:59.514044 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:46:59.514014 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44_5ae819cc-7f64-4b4a-8559-1bc35afe67d1/istio-proxy/0.log" Oct 30 18:47:00.334590 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:00.334559 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jpjk4_c9e23597-79e7-4a28-ab52-fc8b343226ab/dns/0.log" Oct 30 18:47:00.389668 ip-10-0-130-84 kubenswrapper[4455]: E1030 18:47:00.389603 4455 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvc4q" containerName="extractor" Oct 30 18:47:00.450262 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:00.450235 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jpjk4_c9e23597-79e7-4a28-ab52-fc8b343226ab/kube-rbac-proxy/0.log" Oct 30 18:47:00.530635 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:00.530602 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44_5ae819cc-7f64-4b4a-8559-1bc35afe67d1/istio-proxy/0.log" Oct 30 18:47:00.566959 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:00.566933 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bs5gk_e1397e4e-e615-4954-8486-af559b65687e/dns-node-resolver/0.log" Oct 30 18:47:00.846928 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:00.846892 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44_5ae819cc-7f64-4b4a-8559-1bc35afe67d1/istio-proxy/0.log" Oct 30 18:47:01.999675 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:01.999643 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7c94ff96f7-5ln4b_423e7d1a-291e-4960-849f-3307d4293b4e/registry/0.log" Oct 30 18:47:02.453426 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:02.453346 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k75vq_75cac6ab-2f87-4813-80a1-cc6efb29d187/node-ca/0.log" Oct 30 18:47:03.283163 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:03.283111 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-9zh44_c4fd1169-a9c3-4c6d-ba92-1a882c7309b0/discovery/0.log" Oct 30 18:47:03.469085 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:03.469046 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-nrbjr_d427f823-1810-42b1-b753-1790622bca7f/istio-proxy/0.log" Oct 30 18:47:03.606429 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:03.606346 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p_877f995b-cecd-4842-ba49-963b5c1f0590/istio-proxy/0.log" Oct 30 18:47:03.680870 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:03.680841 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-znczm_73b20a69-5907-4bec-a4c6-6b31b46231c4/payload-pre-processing/0.log" Oct 30 18:47:03.750637 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:03.750600 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-9l4x8_909b1ff6-6d4a-4887-9b59-efe2806ac0e8/payload-processing/0.log" Oct 30 18:47:04.443042 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:04.443006 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44_5ae819cc-7f64-4b4a-8559-1bc35afe67d1/istio-proxy/0.log" Oct 30 18:47:04.836930 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:04.836893 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-9zh44_c4fd1169-a9c3-4c6d-ba92-1a882c7309b0/discovery/0.log" Oct 30 18:47:05.007983 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:05.007947 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-nrbjr_d427f823-1810-42b1-b753-1790622bca7f/istio-proxy/0.log" Oct 30 18:47:05.132526 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:05.132439 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p_877f995b-cecd-4842-ba49-963b5c1f0590/istio-proxy/0.log" Oct 30 18:47:05.158714 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:05.158688 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-9zh44_c4fd1169-a9c3-4c6d-ba92-1a882c7309b0/discovery/0.log" Oct 30 18:47:05.194730 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:05.194705 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-znczm_73b20a69-5907-4bec-a4c6-6b31b46231c4/payload-pre-processing/0.log" Oct 30 18:47:05.260793 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:05.260760 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-9l4x8_909b1ff6-6d4a-4887-9b59-efe2806ac0e8/payload-processing/0.log" Oct 30 18:47:05.339798 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:05.339765 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-nrbjr_d427f823-1810-42b1-b753-1790622bca7f/istio-proxy/0.log" Oct 30 18:47:05.491038 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:05.491000 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p_877f995b-cecd-4842-ba49-963b5c1f0590/istio-proxy/0.log" Oct 30 18:47:05.555946 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:05.555910 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-znczm_73b20a69-5907-4bec-a4c6-6b31b46231c4/payload-pre-processing/0.log" Oct 30 18:47:05.619634 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:05.619596 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-9l4x8_909b1ff6-6d4a-4887-9b59-efe2806ac0e8/payload-processing/0.log" Oct 30 18:47:07.183094 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:07.183063 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-9zh44_c4fd1169-a9c3-4c6d-ba92-1a882c7309b0/discovery/0.log" Oct 30 18:47:07.526703 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:07.526675 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-nrbjr_d427f823-1810-42b1-b753-1790622bca7f/istio-proxy/0.log" Oct 30 18:47:07.716980 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:07.716947 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6f89d9f6ccrln44_5ae819cc-7f64-4b4a-8559-1bc35afe67d1/istio-proxy/0.log" Oct 30 18:47:07.771529 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:07.771494 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p_877f995b-cecd-4842-ba49-963b5c1f0590/istio-proxy/0.log" Oct 30 18:47:07.889199 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:07.889121 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-znczm_73b20a69-5907-4bec-a4c6-6b31b46231c4/payload-pre-processing/0.log" Oct 30 18:47:08.003676 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:08.003643 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-9l4x8_909b1ff6-6d4a-4887-9b59-efe2806ac0e8/payload-processing/0.log" Oct 30 18:47:09.557912 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:09.557782 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-75d56799cb-n8rw5_9eea1448-1cc7-4ff1-b584-2b7f3e1b730d/authorino/0.log" Oct 30 18:47:09.604928 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:09.604880 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-779bff956b-bmwkw_8e4d9f9b-bd72-4d8b-a533-79334334789f/manager/0.log" Oct 30 18:47:09.926264 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:09.926174 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-57f5c66455-l7hkg_64c72761-8ae7-47a1-aa47-d10cadeb3e34/manager/1.log" Oct 30 18:47:09.926467 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:09.926273 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-57f5c66455-l7hkg_64c72761-8ae7-47a1-aa47-d10cadeb3e34/manager/0.log" Oct 30 18:47:10.115255 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:10.115218 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6b575997cc-27c49_2cb9c001-fc99-4e94-908f-e3a59edeb3ab/manager/0.log" Oct 30 18:47:10.116712 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:10.116681 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-75d56799cb-n8rw5_9eea1448-1cc7-4ff1-b584-2b7f3e1b730d/authorino/0.log" Oct 30 18:47:10.167833 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:10.167805 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-779bff956b-bmwkw_8e4d9f9b-bd72-4d8b-a533-79334334789f/manager/0.log" Oct 30 18:47:10.504720 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:10.504685 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-57f5c66455-l7hkg_64c72761-8ae7-47a1-aa47-d10cadeb3e34/manager/1.log" Oct 30 18:47:10.505602 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:10.505586 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-57f5c66455-l7hkg_64c72761-8ae7-47a1-aa47-d10cadeb3e34/manager/0.log" Oct 30 18:47:10.687595 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:10.687559 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6b575997cc-27c49_2cb9c001-fc99-4e94-908f-e3a59edeb3ab/manager/0.log" Oct 30 18:47:10.693193 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:10.693170 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-9zh44_c4fd1169-a9c3-4c6d-ba92-1a882c7309b0/discovery/0.log" Oct 30 18:47:10.944263 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:10.944182 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-nrbjr_d427f823-1810-42b1-b753-1790622bca7f/istio-proxy/0.log" Oct 30 18:47:10.946312 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:10.946294 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-p96b8_60c3d534-02fc-4665-ada2-e56fba531b87/serve-healthcheck-canary/0.log" Oct 30 18:47:11.073925 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:11.073897 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-67dd9d5b47-l6g6p_877f995b-cecd-4842-ba49-963b5c1f0590/istio-proxy/0.log" Oct 30 18:47:11.133363 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:11.133333 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-znczm_73b20a69-5907-4bec-a4c6-6b31b46231c4/payload-pre-processing/0.log" Oct 30 18:47:11.195384 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:11.195286 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-9l4x8_909b1ff6-6d4a-4887-9b59-efe2806ac0e8/payload-processing/0.log" Oct 30 18:47:12.347709 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:12.347680 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jvc4q_b38a6e02-c51f-455b-b5bd-25ca428cddb2/kube-rbac-proxy/0.log" Oct 30 18:47:12.459289 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:12.459265 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jvc4q_b38a6e02-c51f-455b-b5bd-25ca428cddb2/exporter/0.log" Oct 30 18:47:12.570379 ip-10-0-130-84 kubenswrapper[4455]: I1030 18:47:12.570352 4455 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jvc4q_b38a6e02-c51f-455b-b5bd-25ca428cddb2/extractor/0.log"